North Carolina AHEC Program
summer 2009 newsletter | home
Charlotte AHEC: Evaluating For Program Quality and Effectiveness
By Julie Covington, MS, RD, LDN,
assistant director, Allied & Public Health Education, Performance Measurement,
Charlotte AHEC. (Reprinted from the Spring 2009 National AHEC Bulletin.)
As today’s health care organizations focus on quality care and improved patient outcomes, quality staff training has become essential in keeping staff knowledge and skills updated. Coupled with our economically challenged environment, ensuring the quality and effectiveness of continuing professional education (CPE) programs is vital as employers face tighter budgets for CPE and limited staff time available for travel. With financial and time constraints, healthcare professionals must make careful decisions about which programs will effectively update their technical skills and in turn improve outcomes for their patients. AHECs are known for providing quality, timely and cost effective training opportunities for health care professionals in their regional settings. With increased need to prove effectiveness through outcome and performance measures, AHECs need to examine their program evaluation methods to ensure they are obtaining the data needed to document that their programs are effective and appropriately targeted to current workforce needs. In 2008 Charlotte AHEC established a Program Evaluation Task Force to assess the effectiveness of our program evaluation process, with a focus on documentation of learning outcomes and program effectiveness. Consistency in program evaluation processes across disciplines was also a priority. This article provides details of our progress and plans for future work in this area.
The Program Evaluation Task Force was established in March 2008, with representation from each AHEC discipline and involving staff at all levels of the program evaluation process, from project coordinators to discipline directors. An initial overview of our program evaluation processes revealed a predominance of Level I, or participant reaction type program, evaluations using a Likert scale for participant satisfaction ratings. There were some inconsistencies between disciplines in how data was summarized and reported, which led to difficulty in comparing program scores between disciplines. A few disciplines, such as leadership development, quality initiatives, and continuing medical education, had begun to implement more in-depth evaluation processes which assessed learning outcomes. Led by Joann Spaleta, director of allied health, public health, leadership development and health careers for Charlotte AHEC, the task force participated in group training on the Kirkpatrick model of program evaluation, utilizing the four levels of assessment as displayed in Table 1. This established a common terminology and framework for task force members to work with in brainstorming ideas for improvements to our current policies and procedures in this area. Additionally, the majority of task force members attended the summer 2008 North Carolina AHEC Academy of Professional and Organizational Development (POD) workshop, Continuing Education: Moving From Reaction to Learning and Beyond. Developed and taught by Larry Freeman, DMin, LCAS, associate director of Northwest AHEC and Joann Spaleta, MBA, MHA, MT(ASCP) of Charlotte AHEC, this one day workshop provided an intensive opportunity to study the most up-to-date evaluation techniques and tools to assess CPE performance and outcomes. Supported by this additional training, Charlotte AHEC staff moved forward in updating program evaluation policies and procedures to ensure consistency across disciplines, began implementing increased levels of program evaluation (as appropriate) across disciplines, and improved documentation of program outcomes.
Currently, Charlotte AHEC utilizes all four levels of Kirkpatrick’s model of program evaluation in assessing program outcomes. All programs begin with a level one (reaction) assessment, measured using a Likert scale. Participant comments on the program setting, speaker, and whether objectives were met are also requested. Basic information on outreach and requested topics for future programs is also obtained. A minimum standard of 4.0 on a 5.0 scale was set to determine program success on this basic level of evaluation. If a program overall score falls below 4.0, staff will determine recommendations for improvement with documentation in the program file.
Level two evaluation (learning) is currently being implemented in several Charlotte AHEC programs, with a goal of at least 40% program evaluations being completed at this level. One example is the Medical Terminology Made Easy course. Students take a pretest the first day of class and a post test at the final (sixth) session. Comparison of course pre and post tests have allowed us to document an average of 48% improvement in student knowledge of medical terminology at the end of the six week program. Another example is the Total Parenteral Nutrition (TPN) Update program. During the last hour of the program, participants are given case studies and asked to develop TPN prescriptions for the case study patients. When finished, instructors review and discuss the answers in class, allowing participants the opportunity to assess their knowledge and catch areas they missed.
Level three evaluation (application) is currently being completed for several Charlotte AHEC conferences and workshops, with the goal of at least 10% of our programs being evaluated in this manner. This level assesses the participant’s ability to apply their training in the work setting. Evaluating programs at this level can be very useful in providing program outcome data. Participants are asked to provide information on skill or knowledge application in the workplace approximately three to six months after instruction. From this data, instructors can determine which elements of the program were learned sufficiently for workplace application; if workplace application has not occurred the instructors can hone in on why and make necessary improvements to future programs. This is usually achieved using electronic surveys sent to program participants three to six months after attending a targeted conference or workshop.
Level four evalaution (results) is more time consuming, however the information obtained can be invaluable in documenting a program’s ultimate impact on organization performance. Not all programs are appropriate for this level of evaluation; at Charlotte AHEC our goal is to evaluate 5% of our programs in this manner. To be a candidate for this level of evaluation, the program must be able to produce measurable results in the work setting. One example of this evaluation level taking place at Charlotte AHEC is our Quality Improvement 101: A Toolbox for Improvement course. During this two part course, participants form teams to plan and implement a quality improvement project in their workplace. Between sessions, three one-hour conference calls and a listserv help to maintain team momentum. In the second session, participants share project outcomes and develop strategies to maintain and possibly replicate project successes.
Targeted and effective program evaluation is essential for providing AHECs with the data they need to improve program quality and prove program effectiveness. Recent improvements in Charlotte AHEC’s program evaluation processes are already helping to better document positive program outcomes. Due to interest in our region, a further program evaluation course is in development at Charlotte AHEC for our Fall 2009 calendar.
References: Freeman, L., & Spaleta, J. (2008). Evaluating CE: Moving from reaction to learning and beyond. AHEC Academy of Professional and Organizational Development, North Carolina Area Health Education Centers Program. Kirkpatrick, D.L., & Kirkpatrick, J.D. (2006). Evaluating Training Programs: The Four Levels (3rd ed.). San Francisco: Berrett-Koehler Publishers, Inc.
________________
TABLE 1. Kirkpatrick’s Four Levels of Evaluation Level (and What the Level Measures*)
- Reaction: Was the participant satisfied with the educational program?
- Learning: What did the participant learn from the educational program in the classroom?
- Application/Behavior Change: How did the educational program affect the participant performance in the workplace?
- Results: Did improvements in the participant’s performance attributed to the educational program influence the organizational performance?
*Measuring the effectiveness of educational programs
Reprinted with permission from Joann Spaleta, MBA, MHA, MT(ASCP) of Charlotte AHEC from Continuing Education: Moving From Reaction to Learning and Beyond program materials, based on information from Evaluating Training Programs: The Four Levels (3rd ed.) by D.L. Kirkpatrick & J.D. Kirkpatrick (2006).
________________
Recommendations for Further Reading:
Barksdale, S., & Lund, T. (2001). Rapid Evaluation. Virginia: ASTD Press
Kirkpatrick, D.L., & Kirkpatrick, J.D. (2006). Evaluating Training Programs: The Four Levels (3rd ed.). San Francisco: Berrett-Koehler Publishers, Inc.
Kirkpatrick, D.L., & Kirkpatrick, J.D. (2007). Implementing The Four Levels. San Francisco: Berrett-Koehler Publishers, Inc. Author: Julie Covington, MS RD LDN Assistant Director, Allied & Public Health Education, Performance Measurement Charlotte AHEC December 2008