UNC Anesthesiology faculty and UNC research collaborators have co-authored a multi-disciplinary June 2025 Anesthesia & Analgesia article comparing generative A.I. and human test performance on the American Board of Anesthesiology (ABA)’s Standardized Oral Examination (SOE). This study is a first-of-kind exploration of whether a language-learning model generative A.I. tool like ChatGPT can produce acceptable and reasonable responses to questions within higher-level SOE competencies, as compared with human exam takers. UNC Anesthesiology authors included: Drs. Samuel Blacker, Fei Chen, Harendra Arora, Anthony (Tony) Passannante, David Zvara, Ben Cobb, Alexander Doyal, Daniel Rosenkrans, Brad Brown, Michael Gonzales, Courtney Hood, Tiffany Pham and Robert Isaak.
Entitled, “An Exploratory Analysis of ChatGPT Compared to Human Performance With the Anesthesiology Oral Board Examination: Initial Insights and Implications,” this randomized study was designed to produce evidence on the usefulness of generative A.I. tools to aid fellows preparing to take the SOE exam as a post-residency, crucial component of the initial certification process.
In this study, four UNC board-certified anesthesiologists, none of whom are ABA examiners, served as blinded examiners. A total of six SOE practice exams were administered to anesthesiology fellows preparing to take the ABA SOE. By comparison, two sample, online ABA examinations were fed into ChatGPT. In study findings, human test takers had higher median scores than ChatGPT on specific topics, yet both showed no significant difference in overall global scores. Though ChatGPT showed no issues with factual accuracy, examiners deemed the generative A.I. tool’s responses to be consistently lengthy and lacking in focus. Study authors concluded their results justified further exploration of training an A.I. database like ChatGPT to produce responses more applicable to ABA SOE questions. They noted study findings additionally support future training of generative A.I. platforms like ChatGPT to serve as anesthesiology-specific, A.I.-trained SOE preparation tools designed to support improved SOE performance in practice exam takers.
Associate Professor of Anesthesiology, UNC Medical Center Neuroanesthesia Section Chief and lead study author Sam Blacker, MD, was interviewed by Anesthesia & Analgesia on its podcast OpenAnesthesis.org. (scroll to June 2025 episode). Yale School of Medicine Professor of Anesthesiology Robert Gaiser, MD, penned a corresponding editorial in A Beautiful Tomorrow in the World of Graduate Medical Education and Assessment. To read the full text of the study authored by Dr. Blacker and colleagues, click HERE.