Certain Accreditation Council for Graduate Medical Education (ACGME) milestone competencies can be challenging to assess in clinical practice. Additionally, the American Board of Anesthesiology (ABA) rolled out an Objective Standardized Clinical Exams (OSCE) as part of the Applied Examination for primary certification. As a result, we developed an OSCE for our anesthesiology residency program to evaluate these difficult to assess milestones while also providing an opportunity to prepare for the ABA OSCE. One of the main focuses of our simulation-based research projects is to develop best practices regarding implementation and assessment of OSCEs for anesthesiology learners.

Validity of Simulation-Based Milestones Assessment

Related publications and news reports

Validity of Simulation-Based Milestones Assessment

Anesthesiology residency programs bi-annually report resident Accreditation Council for Graduate Medical Education (ACGME) milestones to the American Board of Anesthesiology. Some milestone domains are particularly challenging to reliably observe during clinical care. Simulation-Based Milestones Assessment (SBMA) may help overcome these challenges. However, few studies have examined the external validity of simulation assessment scores (i.e., the relationships between simulation-based assessment scores and other standard measures of ability) for milestones. This study analyzed whether SBMA scores (1) discriminate by postgraduate year, (2) improve over time, and (3) correlate with traditional measures of performance. The associations between SBMA scores and experience level, time-in-training, and clinical performance evaluations provide evidence that SBMA may be used as metrics of residents’ Accreditation Council for Graduate Medical Education milestone competencies.

Isaak, R, Martinelli, S, Chen, F, Arora, H, Zvara, D, Hobbs, G, Stiegler, M. Validity of simulation-based assessments for ACGME milestone achievement. Simulation in Healthcare. 2018 June; 13(3):201-210

Related publications and news reports

University of North Carolina at Chapel Provost’s Hill IBM Junior Faculty Development Award of $7,500 for the project titled: Simulated Clinical Scenarios for ACGME Milestones: Live-In Person Versus Delayed Video Recorded Assessments. Date: 12/01/14-11/20/15. PI: Robert Isaak.

Isaak, R., Chen, F., Arora, H., Martinelli, S., Zvara, D., Stiegler, M., How are Anesthesiology Programs Preparing Residents for the new APPLIED Certification Exam? A Descriptive Survey of Residency Simulation Programs. Anesthesia and Analgesia, 2017 Sep 125(3):991-998

Isaak R, Chen, F., Hobbs, G., Martinelli, S., Stiegler, M., Arora, H. Standardized Mixed-Fidelity Simulation for ACGME Milestones Competency Assessment and Objective Structured Clinical Exam Preparation. Medical Science Educator. 2016, Sept. 26 (3):437-441

Isaak R, Stiegler M, Hobbs G, Martinelli S, Zvara D, Arora H, Chen F, Comparing Real-time Versus Delayed Video Assessments for Evaluating ACGME Sub-Competency Milestones in Simulated Patient Care Environments. Cureus. 2018 Mar 4;10(3): e2267.

Rebel, A., DiLorenzo, A., Isaak, R., McGrane, S., Moran, K., Mobley, E., Schell, R., Replicating an Educational OSCE Project for Skill Assessment of Junior Anesthesiology Residents from One Institution to Others: A Qualitative Description. Journal of Education in Perioperative Medicine. 2018 Apr;20(2);1-12

Chen, F., Carter, T., Maguire, D., Blanchard, E., Martinelli, S., Isaak, R., Experience is the Teacher of All Things: Prior Participation in Anesthesiology OSCEs Enhances Communication of Treatment Options with Simulated High-Risk Patients. Journal of Education in Perioperative Medicine. 2019 July 1;21(3):E626

Fei Chen, Sania Rahim, Rob Isaak, Brooke Chidgey, Emily Teeter, Harendra Arora, and Susan M. Martinelli. 2023. Maintaining Social Support in the Era of Social Distancing: Transitioning an in-person family-oriented wellness event to a virtual venue. eLearn 2023, Special Issue, Article 4 (February 2023). https://doi.org/10.1145/3583062.3539612

Contact TEACHER Lab

Department of Anesthesiology
The University of North Carolina at Chapel Hill
Phone 919-966-5136
Email: Anes_TEACHER_Lab@med.unc.edu