AI Breakthrough: ChatGPT can almost pass US Medical Licensing Exam, study finds

Artificial intelligence-powered platform ChatGPT can nearly pass the US Medical Licensing Exam (USMLE), a new study found.

The study, published in the journal PLOS Digital Health, tested ChatGPT’s performance on the USMLE, a series of three exams that assess medical knowledge in various disciplines, such as biochemistry, diagnostic reasoning, and bioethics.

The test is required for medical licensure in the US and is taken by medical students and physicians-in-training.

After removing image-based questions and indeterminate responses, ChatGPT scored between 52.4 percent and 75.0 percent across the three USMLE exams. The passing threshold for the exam is approximately 60 percent.

ChatGPT outperformed PubMedGPT, a counterpart model trained exclusively on biomedical domain literature, which scored 50.8 percent on an older dataset of USMLE-style questions.

The study’s authors Tiffany Kung and Victor Tseng noted that their findings provide a glimpse of ChatGPT’s potential to improve medical education and eventually, clinical practice. For example, clinicians at AnsibleHealth are already using ChatGPT to simplify jargon-heavy reports for easier patient comprehension.

ChatGPT is a large language model designed to generate human-like text by predicting word sequences, but unlike most chatbots, it cannot search the internet. Instead, it uses internal processes to generate text.

The authors of the study stated that “reaching the passing score for this notoriously difficult expert exam, and doing so without any human reinforcement, marks a notable milestone in clinical AI maturation.”

“ChatGPT contributed substantially to the writing of [our] manuscript… We interacted with ChatGPT much like a colleague, asking it to synthesize, simplify, and offer counterpoints to drafts in progress…All of the co-authors valued ChatGPT’s input,” said Dr. Kung in a statement on Thursday, highlighting that ChatGPT’s role in this research went beyond being the study subject.

The study demonstrates the potential for AI to improve medical education and eventually, clinical practice. With its ability to score at or around the passing threshold for the USMLE, ChatGPT represents a milestone in the maturation of clinical AI.

Since it was launched in November last year, ChatGPT has garnered global attention due to its ability to simplify a variety of tasks. It is available 24/7 and provides instant responses, making it a convenient tool for those seeking quick answers or help with somewhat complicated tasks.

ChatGPT’s versatility allows it to perform a variety of tasks, such as answering questions, generating text, or carrying out a conversation, making it a flexible tool for a range of purposes.