Assessing the Capabilities of ChatGPT-4 in Accurately Responding to Frequently Asked Questions Derived from FDA Regulatory Guidelines: A Detailed Evaluation of Accuracy and Relevance in CMC Applications

Artificial Intelligence (AI) and Machine Learning (ML) are transforming various industries, and their application in regulatory contexts is particularly promising. These technologies offer the potential to enhance efficiency, accuracy, and decision-making processes in regulatory affairs, including Chemistry, Manufacturing, and Controls (CMC)¹. This study investigates the accuracy and reliability of ChatGPT-4 in addressing regulatory questions by comparing its responses to those provided by the FDA in the document “Guideline Changes to an Approved NDA or ANDA: Questions and Answers.” The research aims to assess how effectively ChatGPT-4 aligns with official regulatory guidance, highlighting its potential and limitations in supporting CMC professionals.

AI and ML are increasingly adopted in regulatory affairs to streamline and optimize various processes. One of the primary applications is in data management and analysis. Regulatory bodies and pharmaceutical companies handle immense amounts of data, including clinical trial results, manufacturing records, and compliance documentation. AI and ML algorithms have the capability to process and analyze this data faster and more accurately than traditional methods². This capability is crucial for identifying trends, detecting anomalies, and ensuring compliance with regulatory standards.

AI and ML are also used to enhance risk management in regulatory affairs. These technologies can identify potential risks in the manufacturing process, such as deviations from standard operating procedures or quality control issues³. By detecting these risks early, companies can take corrective actions before they escalate into significant compliance problems. This proactive approach to risk management helps ensure the safety and efficacy of pharmaceutical products, which is a primary concern for regulatory agencies.

Furthermore, evaluating AI's performance in regulatory contexts can highlight areas where the model may need further fine-tuning or improvements to ensure its responses are consistent with official standards⁴. This is especially crucial when integrating AI into workflows involving compliance, documentation, and reporting. By assessing AI accuracy in regulatory environments, companies can more confidently leverage these technologies, knowing they are meeting regulatory requirements and minimizing risks associated with non-compliance⁵. In the context of CMC, AI and ML can automate routine tasks such as data collection, reporting, and documentation. For example, AI-powered systems can automatically generate reports that meet the specific requirements of regulatory agencies, reducing the time and effort required from human professionals⁶. This automation not only increases efficiency but also minimizes the risk of human error, which can have significant implications for regulatory compliance.

Introduction to ChatGPT-4 and its capabilities.

ChatGPT-4, the latest iteration of OpenAI’s language model, represents a significant advancement within the realm of artificial intelligence⁷. Built on the GPT-4 architecture, this model is designed to understand and generate human-like text based on the input it receives. Its capabilities extend far beyond simple text generation, making it a versatile tool for a wide range of applications⁸.

A notable characteristic of ChatGPT-4 is its proficiency in maintaining coherent and contextually appropriate conversations. It can understand complex queries, provide detailed responses, and even keep track of the flow of a conversation over multiple interactions. ChatGPT-4 is also adept at generating creative content⁸. Whether it’s writing essays, crafting stories, or even composing poetry, the model can produce high-quality text that often rivals human creativity. This capability is particularly valuable for content creators, marketers, and educators who need to produce substantial amounts of engaging material.

Beyond its creative prowess, ChatGPT-4 excels in technical and specialized domains. It can assist with coding by generating code snippets, debugging errors, and explaining complex programming concepts. For professionals in fields like medicine, law, and finance, ChatGPT-4 can provide insights, summarize documents, and even help draft reports and technical documents⁹. Another significant capability of ChatGPT-4 is its ability to learn and adapt. It can be fine-tuned with specific datasets to understand better and respond to niche topics¹⁰. This adaptability ensures the model remains relevant and applicable across a wide range of industries and applications.

ChatGPT-4 is a versatile and powerful AI model that excels in comprehending and producing human-like text¹¹. Its applications are vast, ranging from customer service to creative writing and technical support, making it a valuable asset in today’s digital landscape.

What is FDA and what are FDA guidelines?

The U.S. Food and Drug Administration (FDA) is an agency of the United States Department of Health and Human Services. Its mission is to safeguard public health by guaranteeing the safety, effectiveness, and security of drugs, biological products, medical devices, food supplies, cosmetics, and radiation-emitting products. Established in 1906, the FDA is crucial in regulating and supervising these products to ensure they meet stringent safety standards¹².

The FDA plays a significant role in public health by regulating various products. It ensures the safety and efficacy of drugs through rigorous clinical trials and post-marketing surveillance. Medical devices are classified into three categories: premarket approval or clearance and ongoing performance monitoring¹³. The FDA sets standards in food safety, conducts inspections, and works to prevent foodborne illnesses. For biologics such as vaccines and gene therapies, the FDA ensures safety, potency, and purity, often collaborating with other agencies. It oversees cosmetics to ensure they are not unsafe or misbranded and regulates radiation-emitting products by setting exposure standards and conducting inspections.

The FDA issues comprehensive guidelines to help manufacturers and stakeholders navigate regulatory requirements, procedures, and standards for various products. These guidelines cover critical areas such as clinical trials, where the FDA provides recommendations on design, conduct, and reporting to ensure reliable and robust data while addressing patient selection, trial protocols, and ethical considerations¹⁴. Additionally, the FDA outlines Good Manufacturing Practices (GMP) for the manufacturing, processing, and packaging of drugs, medical devices, and food products, ensuring consistent quality and compliance with established standards¹⁵.

In conclusion, The FDA is essential in maintaining the safety and effectiveness of a wide range of products that impact public health. Its guidelines and regulations are essential for maintaining high standards and protecting consumers¹⁶. The agency's work is crucial in fostering public trust and advancing medical and scientific innovation.

Importance of evaluating AI accuracy in regulatory settings.

In regulatory settings, accuracy is paramount. Incorrect interpretations or errors in regulatory responses can result into expensive setbacks, compliance failures, or even regulatory penalties. Given that the healthcare and pharmaceutical industries are highly regulated, regulatory affairs professionals must ensure that all interactions, reports, and documentation align with the guidelines established by authorities like the FDA¹⁷. Thus, the ability of AI models like ChatGPT-4 to deliver precise and reliable answers to regulatory questions is critical.

Evaluating AI’s accuracy in these settings helps ensure that AI tools are trustworthy, capable of supporting professionals in making well-informed decisions, and can effectively supplement or streamline the regulatory process. For example, AI can potentially reduce the workload on human experts by automating certain aspects of regulatory affairs, such as answering FAQs or drafting documents¹⁸. However, it must demonstrate a high level of accuracy to be reliable in such sensitive environments.

Another important application of AI and ML in regulatory contexts is predictive analytics. By scrutinizing historical data, AI and ML models can predict potential regulatory hurdles and suggest proactive measures to address them. This predictive capability is particularly valuable in the pharmaceutical industry, where delays in regulatory approval can be costly and time-consuming¹⁹. For instance, AI algorithms can predict the likelihood of a drug passing regulatory scrutiny based on its clinical trial data, allowing companies to make informed decisions about their development strategies.

In conclusion, the accuracy of AI models like ChatGPT-4 in regulatory affairs is essential to ensuring that these tools are effective, reliable, and safe for widespread use. This study aims to provide valuable insights into the potential of AI and understand its readiness for use in regulatory environments by assessing the accuracy and reliability of the responses.

METHODOLOGY

A new email account was established to mitigate any potential negative bias from search algorithms. ChatGPT-4²⁰ application was accessed via a Google search. The 'Continue with Google' option was selected on the login screen. To simulate a standard user journey, the most commonly utilized search engine globally was chosen. Prior to the Question & Answer session, the computer’s search history & cookies were deleted. This was done through the 'Clear Browsing Data' section under the 'Privacy and Security' tab in the search engine's 'Settings' menu. The checkboxes for 'Cached Images and Files,' 'Cookies and Other Site Data,' and 'Browsing History' were selected, and the 'Time Range' was set to match the study period before clearing the data.

Method for FDA Guideline and Question Selection

As a component of the study, an FDA guideline, “Guidance for Industry Changes to an Approved NDA or ANDA Questions and Answers²¹,” document was selected to be analyzed. This document provides questions and answers relating to the guidance on Changes to an Approved NDA or ANDA. The questions are based on those posed to CDER by applicants. The questions and answers are presented using seven subject headings corresponding to the table of contents in the guidance. One question from each subject heading (a total of seven questions) was selected and analyzed. The rest of the questions were not included in the study. Following are the seven questions corresponding to each subject heading (mentioned in parentheses) considered in the study:

1. For a change that is reported in a Supplement - Changes Being Effected in 30 Days, will CDER complete the review of the supplement within 30 Days?²¹ (Reporting categories)
2. A manufacturing change is planned that can be reported in the annual report. Should the data to support the change be included in the annual report?²¹ (General Requirements)
3. Should a prior approval supplement be submitted for a change in the sterilization site for a primary packaging component of a metered dose inhaler?²¹ (Manufacturing sites)
4. Can changes in mixing steps and elimination of a mixing step be reported in an annual report if these changes are implemented prior to the manufacture of validation batches?²¹ (Manufacturing Process)
5. What reporting category should be used if a USP HPLC assay procedure replaces, or is used in addition to, a microbiological assay that is listed in the approved specification as the regulatory analytical procedure?²¹ (Specifications)
6. The plastic used in a desiccant canister is being changed. When the desiccant is used for bottles of solid oral dosage form products, should it be reported as a Supplement - Changes Being Effected under section IX.C.2.b of the guidance?²¹ (Package)
7. How should the addition of a test to an approved stability protocol be reported?²¹ (Miscellaneous Changes)

Testing protocol

ChatGPT-4 utilizes a comprehensive language model to interpret user inquiries on various subjects. In order to assess the reliability and precision of its responses, a comparative study was conducted using seven carefully selected questions.

These seven questions were presented to ChatGPT-4 in their original form, ensuring that they remained unchanged and unmodified to preserve the integrity of the inquiry. The questions were presented consecutively, without any interruptions, while recording the responses in English. This approach allowed for a systematic comparison of how the model handled each question while maintaining a consistent format throughout the process. By presenting the questions in a single, uninterrupted text message, the study ensured that the context remained clear for the model, providing a controlled environment for analyzing the responses. This protocol enabled the assessment of the accuracy, relevance, and quality of the answers provided by ChatGPT-4 to these seven specific inquiries.

Evaluation Criteria

The evaluation of the responses given by ChatGPT-4 for the chosen seven questions, compared with the FDA guidelines answers, was conducted through a detailed analysis of several key criteria. These criteria include Accuracy, Relevance, Completeness, Consistency, and Commonality²². The researchers assessed the Accuracy of ChatGPT-4's responses by comparing factual correctness and alignment with the official FDA guidelines. The Relevance of the responses was measured by how well the answers addressed the specific regulatory concerns outlined in the questions. Completeness was evaluated by examining whether the answers covered all necessary aspects of the guidelines without omitting critical information. Consistency was analyzed by checking if ChatGPT-4's responses remained stable and coherent across different questions or contexts and whether they aligned consistently with the FDA's guidance²³. Additionally, the researchers identified Core Ideas, which represent the main concepts or points conveyed in the answers, as well as Common Ideas, which are similar points between the FDA’s answers and ChatGPT-4’s responses. Inconsistent ideas were also flagged, focusing on any discrepancies or contradictions between the two sources²⁴. A quality analysis of the answers was conducted to evaluate the overall depth, clarity, and usefulness of the answers given by ChatGPT-4, offering insights into its effectiveness as a tool in regulatory decision-making²⁵. These criteria were used to test the quality of ChatGPT-4’s responses and highlight both its strengths and limitations in accurately reflecting the FDA’s regulatory framework²⁶. Below are the criteria questions to test the above topics:

1. Relevance

1a. Does the response capture the central concept of the question without going off-track?

2. Accuracy

2a. Is any part of the provided answer/information factually, correct?

2b. Are there any inaccurate or deceptive information/statements provided?

3. Completeness

3a. Is the language of the answer clear, concise, and easy to comprehend?

3b. Is there any complex terminology/language used in the answer?

4. Consistency and Commonality

4a. Are there any shared concepts between the answers provided by FDA guideline and ChatGPT?

4b. Are there any inconsistencies between the answers provided by FDA guideline and ChatGPT?

Chat GPT-4 Answers

Q1. For a change that is reported in a Supplement - Changes Being Effected in 30 Days, will CDER complete the review of the supplement within 30 Days? ²¹

Ans. A Changes Being Effected in 30 Days (CBE-30) supplement submitted to the Center for Drug Evaluation and Research (CDER) allows the sponsor to implement the proposed change after 30 days, provided the FDA does not object during that time. However, the 30-day period does not guarantee that CDER will complete its review of the supplement within this timeframe. The review process may extend beyond 30 days, depending on the complexity of the change and CDER’s workload. While the sponsor may proceed with the change after the 30-day period, they do so at their own risk. If CDER subsequently determines that the change is not acceptable, the sponsor may be required to reverse the change or take corrective actions.

Q2. A manufacturing change is planned that can be reported in the annual report. Should the data to support the change be included in the annual report? ²¹ (General Requirements)

Ans. Including data to support a planned manufacturing change in an annual report depends on the relevance and materiality of the information to stakeholders. If the change significantly impacts financial performance, operational efficiency, environmental sustainability, or product quality, it is beneficial to include supporting data to provide transparency and context. Stakeholders such as shareholders and regulators may be particularly interested if the change aligns with broader goals like cost reduction, innovation, or sustainability. Additionally, certain industries or regions may have regulatory requirements that mandate disclosure of planned changes affecting the company’s risk profile, compliance, or operations. When including such information, it is important to maintain the report’s focus and strategic tone. A high-level summary of the purpose and expected outcomes of the change, along with key metrics or benefits, is usually sufficient for the main report. Detailed technical or operational data, if necessary, can be provided in appendices or supplementary documents such as sustainability or investor reports. This approach ensures transparency while keeping the annual report concise and relevant.

Q3. Should a prior approval supplement be submitted for a change in the sterilization site for a primary packaging component of a metered dose inhaler? ²¹ (Manufacturing sites)

Ans. Yes, a Prior Approval Supplement (PAS) is typically required for a change in the sterilization site of a primary packaging component of a metered dose inhaler (MDI). Such changes are considered significant because they can directly impact the sterility, quality, and performance of the product. The FDA categorizes these as major changes, which necessitate regulatory review and approval before implementation. A PAS submission should include a detailed description of the change, including information about the new sterilization site and the rationale for the change. Additionally, it should provide sterilization process validation data to ensure the new process meets regulatory standards, as well as comparative data demonstrating that the new site maintains equivalent sterility assurance without negatively affecting the product’s quality. Evidence of the new site’s compliance with Current Good Manufacturing Practices (cGMP) should also be included. To ensure compliance, manufacturers should refer to FDA guidance documents, such as the “Changes to an Approved NDA or ANDA” guideline.

Q4. Can changes in mixing steps and elimination of a mixing step be reported in an annual report if these changes are implemented prior to the manufacture of validation batches? ²¹ (Manufacturing Process)  

Ans. Changes in mixing steps or the elimination of a mixing step, if implemented prior to the manufacture of validation batches, may often be reported in an annual report, provided they are classified as minor changes and do not significantly affect the drug product’s quality, safety, or efficacy. Regulatory guidelines, such as the FDA’s Changes to an Approved NDA or ANDA or ICH Q10, emphasize the need to evaluate the potential impact of manufacturing changes on product quality. Minor changes, such as slight adjustments in mixing parameters, are typically eligible for reporting in an annual report, while moderate or major changes, such as the elimination of critical steps or significant process alterations, often require prior regulatory approval or notification.

When changes occur before validation batch manufacturing and are supported by adequate risk assessments and data demonstrating no adverse impact on product quality, they are more likely to qualify for annual reporting. It is crucial that validation batch results confirm the changes do not affect the product’s critical quality attributes (CQAs). In the annual report, a detailed description of the change, the rationale behind it, supporting studies showing no impact on quality, and confirmation of implementation before validation batch manufacture should be included.
However, if the changes are significant or introduce uncertainty about the product’s quality, they may require submission of a supplement (CBE-30 or PAS) to the regulatory authority before implementation. Referring to specific regulatory guidelines, such as the FDA’s Scale-Up and Post-Approval Changes (SUPAC) guidance, and consulting with the regulatory affairs team is recommended for a thorough evaluation.

Q.5 What reporting category should be used if a USP HPLC assay procedure replaces, or is used in addition to, a microbiological assay that is listed in the approved specification as the regulatory analytical procedure? ²¹ (Specifications)

Ans. If a USP HPLC assay procedure replaces or is used in addition to a microbiological assay listed in the approved specification as the regulatory analytical procedure, it typically falls under the “Analytical Procedure Changes” reporting category. The specific reporting category depends on the nature and impact of the change. A Prior Approval Supplement (PAS) would be required if the change significantly affects the method’s performance characteristics, such as accuracy, precision, sensitivity, or specificity. This is likely applicable because HPLC and microbiological assays are fundamentally different techniques, and the change could have a significant impact on the regulatory commitment. Alternatively, if the change is less significant and does not adversely affect the product’s quality, safety, or efficacy, it might qualify for a Changes Being Effected (CBE-30 or CBE-0) submission. For instance, this could apply if the HPLC method demonstrates equivalent or superior performance compared to the microbiological assay.

If the HPLC assay is being added alongside the microbiological method without replacing it and does not impact the regulatory specifications, the change might be reportable in the Annual Report. This, however, depends on whether the change affects the overall control strategy or product quality. Regardless of the reporting category, it is essential to provide robust scientific justification, validation data, and comparative studies demonstrating the equivalence or superiority of the HPLC method. Reference to FDA guidance, such as “Changes to an Approved NDA or ANDA” and ICH Q2(R1) for analytical validation expectations, is critical in making this determination. To ensure compliance, consultation with the FDA or the appropriate regulatory authority is recommended to confirm the appropriate reporting category.

Q6. The plastic used in a desiccant canister is being changed. When the desiccant is used for bottles of solid oral dosage form products, should it be reported as a Supplement - Changes Being Effected under section IX.C.2.b of the guidance? ²¹ (Package)

Ans. Yes, changing the plastic used in a desiccant canister for bottles of solid oral dosage form products should typically be reported as a Supplement - Changes Being Effected (CBE) under section IX.C.2 of the FDA guidance on “Changes to an Approved NDA or ANDA”.
This section refers to “container closure systems” and specifies that certain changes to the materials of a packaging system that can affect product quality must be reported as CBE. Since desiccants are part of the container closure system, changing the plastic material may have the potential to impact the moisture protection, stability, or integrity of the drug product. Therefore, a CBE submission is appropriate to ensure the FDA is notified before or shortly after implementation of the change.

However, it is crucial to confirm whether the new plastic material complies with the necessary standards (e.g., USP, FDA requirements) and that it does not adversely impact the drug product’s safety, identity, strength, quality, or purity. Conducting appropriate stability and compatibility studies to support the change is also essential.

Q7. How should the addition of a test to an approved stability protocol be reported^{? 21}  (Miscellaneous Changes)

Ans. The addition of a test to an approved stability protocol must be managed carefully to ensure compliance with regulatory requirements. First, it is essential to assess whether the change constitutes a minor, moderate, or major modification based on its potential impact on product quality, safety, or efficacy. The updated stability protocol should clearly document the new test’s purpose, methodology, and acceptance criteria, along with a rationale for its inclusion. Internally, the revised protocol should be reviewed and approved through the organization’s quality assurance (QA) or quality management system (QMS).
The regulatory reporting requirements for this change will depend on the jurisdiction and regulatory authority overseeing the product. For instance, in the U.S., if the change enhances understanding of stability without altering existing commitments, it might be reported in the next Annual Report. However, significant changes to the protocol may require submission as a Prior Approval Supplement (PAS) or Changes Being Effected (CBE). In the EU, the change could be filed as a Type IA, IB, or II variation depending on its significance. In other regions, the appropriate variation or amendment procedures must be followed. Consulting the relevant regulatory authority may be necessary to confirm the appropriate reporting pathway for significant changes.
After addressing regulatory requirements, the new test can be implemented as part of the updated stability protocol. It is critical to ensure the test complies with pharmacopeial standards or is validated if non-compendial. All related documentation, including justifications, internal approvals, and regulatory submissions, should be retained. Since changes impacting critical quality attributes (CQAs) often face higher scrutiny, careful planning and communication with authorities are key to ensuring a smooth process.

DISCUSSIONS

Table 1: Summary of the Quality Analysis of ChatGPT-4 responses alongside FDA guideline

Table 2: Commonalities and Discrepancies between FDA and Chat GPT Answers