Donald J. Sefcik, DO, MBA Senior Associate Dean College of Osteopathic Medicine Michigan State University
• In-Service Examination (ISE) scores � What do they represent? � How should you interpret them? � What should you do with them? • Techniques used to raise ISE scores • Recommendations
Emergency Medicine • Hern et al (2009); Cheng (2008); Gillen (1997) Family Medicine • Sevensma SC, Navarre G, Richards RK (2008) Internal Medicine • McDonald, Zeger & Kolars (2008) • Garilbaldi et al (2002) OB/Gyne • Withiam-Leitch & Olawaiye (2008) Pediatrics • Aeder, Fogel & Schaeffer (2010) • Langenau, Fogel & Schaeffer (2009) Radiology Surgery • Shellito et al (2010); Subhas et al (2009); Kosir et al (2008) • Ferguson & Warshaw (2006); Virgilio et al (2003); Bull et al (2001)
A 72 year-old patient presents with progressive difficulty breathing, jugular venous distention, orthopnea, bilateral lower extremity edema , and bilateral rales. Which of the following is most important when deciding upon treatment options? A. minute respiration B. heart rate C. degree of lower extremity edema D. respiratory rate E. none of the above
A 72 year-old patient presents with progressive difficulty breathing, jugular venous distention, orthopnea, bilateral lower extremity edema , and bilateral rales. Which of the following is most important when deciding upon treatment options? We don’t treat patients without understanding the etiology of their predicament and the nature of the intervention. Guiding residents’ learning should follow similar guidelines.
X R = X T + M E
X R = X T + M E X R = Reported ISE score X T = True Score M E = Measurement Errors
X R = X T + M E X R = Reported ISE Score Scores Percent = items correct = raw score Percentile = resident rank compared to peers
The report of the performance of a PGY2 on his ISE reveals a percent correct of 63 and a percentile rank of 67. What should you do? A. mandate the resident enter a remediation program B. reduce work hours by 25% to increase study time C. do not promote the resident to the PGY3 year D. encourage the resident to keep studying E. resign as program director
• What is an “average” percent score ? http://www.eric.vcu.edu/home/curriculum/print/2009Histograms.pdf
• What is an “average” percent score ? 2009 IM (n = 402 programs) PGY1 = 54% PGY2 = 59% PGY3 = 63% http://www.eric.vcu.edu/home/curriculum/print/2009Histograms.pdf
• What is an “average” percent score ? 2009 IM (n = 402 programs) PGY1 = 54% PGY2 = 59%(example was PGY2 = 63%) PGY3 = 63% http://www.eric.vcu.edu/home/curriculum/print/2009Histograms.pdf
Your PGY2 from the previous example is now a PGY3. His percentile rank while in your program has been PGY1 = 62, PGY2 = 67, PGY3 = 64. What should you do? realize that this is a common pattern of progression A. worry that this resident will fail his certifying exam B. do not promote the resident to the PGY4 year C. terminate this resident from your program D. delegate the situation to your associate PD E.
Why is percentile important? Increased risk of failing the certifying exam %F 30th 50th 100th
Why is percentile important? Increased risk of failing the certifying exam The lower the percentile rank – The greater the risk of failing the certifying examination %F 30th 50th 100th
Percent Correct Percentile Increases each year PGY1 PGY2 PGY3 PGY4
Percent Correct Percentile Potential Problem PGY1 PGY2 PGY3 PGY4
Percent Correct Percentile Tends to be stable PGY1 PGY2 PGY3 PGY4
Percent Correct Percentile Potential Problem PGY1 PGY2 PGY3 PGY4
Tracking percentiles • Three different residents • Three years of scores C 1 B 1 A 1 %F 30th 50th 100th
Tracking percentiles • Three different residents • Three years of scores C 1 B 1 A 1 C 2 B 2 A 2 %F 30th 50th 100th
Tracking percentiles • Three different residents • Three years of scores C 1 B 1 A 1 C 2 B 2 A 2 B 3 C 3 A 3 %F 30th 50th 100th
Tracking percentiles • Three different residents What can you do • Three years of scores to assess the residents between the annual ISE’s? C 1 B 1 A 1 C 2 B 2 A 2 B 3 C 3 A 3 %F 30th 50th 100th
• Make decisions about • individual’s educational progression • program content areas that are outliers
• Make decisions about • individual’s educational progression • program content areas that are outliers • Do not make decisions about: • promotion or termination
X R = X T + M E X T = True Score = two components
X R = X T + M E X T = True Score = two components X C = Score based on content X B = Score based on beyond content
X R = X C + X B + M E X R = Reported ISE Score X C = Score based on content X B = Score based on beyond content M E = Measurement Error
• Choosing your intervention options • Content • Beyond Content
X C X B + Beyond Content Content PGY1 PGY3 PGY4 PGY2
• Assigned / Programmed Reading • Conference Attendance • Didactic lectures • Grand Rounds / M&M • Practice Questions • Board Review Courses
• Assigned / Programmed Reading • Conference Attendance • Didactic lectures / Journal Clubs • Grand Rounds / M&M • Practice Questions • Board Review Courses
• Self-directed Studying • Preparing Lectures for Peers • Reading Fluency and Accuracy • Test Anxiety
• Self-directed Studying • Preparing Lectures for Peers • Questions from / Clarification by an Attending • Reading Fluency and Accuracy • Developing theme in repeat failures • Test Anxiety • Definite impact on memory / recall
• Intervention options • Content = more impact earlier in training PGY<2 - Programmed Reading / Didactics PGY>3 - Programmed Reading / Didactics
• Beyond Content • Resident Heterogeneity (all years of training) Studying Skills Test-taking Skills
• Beyond Content • Resident Heterogeneity Studying Skills • Memory techniques • Reading and Note-taking • Transfer Test-taking Skills • Reading- Fluency and Accuracy • Test Anxiety • Test-Wiseness
1 / 6
2 / 6
3 / 6
4 / 6 1. Assess the resident’s situation Develop an individualized plan It’s not just about X C 2. Create a team Resident and Attending Attending = Coach to Mentor 3. Establish a schedule Regular meetings 4. Monitor and Refine
5 / 6 1. Assess the resident’s situation Develop an individualized plan There are It’s not just about X C resources 2. Create a team to help you Resident and Attending help your Attending = Coach to Mentor resident 3. Establish a schedule Regular meetings 4. Monitor and Refine
6 / 6 We need to transition away from our focus on teaching (faculty-centered) and instead begin to focus on learning (resident-centered).
Are there some Questions? Emergency Medicine • Hern et al (2009); Cheng (2008); Gillen (1997) Family Medicine • Sevensma SC, Navarre G, Richards RK (2008) Internal Medicine Complete Reference List • McDonald, Zeger & Kolars (2008) • Garilbaldi et al (2002) will be posted (pdf) OB/Gyne on SCS website • Withiam-Leitch & Olawaiye (2008) Pediatrics • Aeder, Fogel & Schaeffer (2010) • Langenau, Fogel & Schaeffer (2009) Radiology Surgery • Shellito et al (2010); Subhas et al (2009); Kosir et al (2008) • Ferguson & Warshaw (2006); Virgilio et al (2003); Bull et al (2001)
Recommend
More recommend
