Practical Strategies for Improving the Frequency and Quality of Direct Observation of Learners Jennifer R. Kogan, MD Professor of Medicine Assistant Dean, Faculty Development Director Undergraduate Education, Department of Medicine Disclosure I have no relevant commercial interests to disclose. 2 Page 1 1
Workplace Based Assessment Assessment of day-to-day practice in the authentic clinical environment Assessment of what doctors actually do in practice Swanwick T. Br J Hosp Med .2009;70:290-3 3 4 Page 2 2
Miller’s Assessment Pyramid Expertise Direct Observation/ WBA Authenticity DOES Standardized patients SHOWS HOW Problem based learning KNOWS HOW Multiple choice exam KNOWS Miller GE. Acad Med. 1990; 65:S63-7 5 Goal To improve the frequency and quality of WBA in your training program 6 Page 3 3
Roadmap Review the rationale for direct observation Identify barriers to frequent, high quality WBA Describe strategies to improve WBA 7 Video Exercise Watch the following encounter between a patient and a PGY-2 resident List areas performed well List errors/deficiencies How would you supervise this resident the next time? Circle a mini-CEX rating for counseling 8 Page 4 4
Roadmap Review the rationale for direct observation Identify barriers to frequent, high quality WBA Describe strategies to improve WBA 9 Rationale for WBA 1. Clinical skills matter but are variable 2. Central in CMBE 3. Essential for deliberate practice 4. Necessary for effective supervision 10 10 Page 5 5
1: Clinical Skills Matter History leads to diagnosis > 80% of time Even in era of technology Required to avoid unnecessary testing Faulty data gathering Source of diagnostic errors Common cause of death Hampton JR et al. BMJ 1975; 2(5969):486-9 Peterson MC et al. West J Med . 1992; 156(2):163-5 Graber, M et al . Acad Med . 2002;77(10):981-92 National Academy of Medicine. Improving Diagnosis in Medicine , 2015 Makary MA et al. BMJ. 2016; 353:i2139 doi: 10.1136/bmj.i2139 11 11 Patient Centered Care Matters Improves communication Promotes patient involvement in care Increases patient knowledge/self-efficacy Creates positive relationships with provider Improves adherence Improves well-being Improves outcomes Decreases costs Levinson W et al. 2010; Health Aff 29: 1310-18 Williams S et al. Fam Prac .1998;15:480-92 DiMatteo M. Patient Educ Counsel . 2004;55:339-44 Stewart M . CMAJ . 1995; 152:1423-33 Vermeir P et al. Int J Clin Pract . 2015;69:1257-1267. 12 12 Page 6 6
Variable Clinical Skills Graduating students/starting interns Practicing physicians Stillman. Ann Intern Med.1990; Sachdeva. Arch Surg.1995; Lypson.Acad Med.2004; Wilson. Acad Med 2002; Fox. Med Educ 2002; Mangione. JAMA1997; Braddock.1999 13 13 Direct Observation to Assess Core Skills ASSESSMENT Legitimizes the skills Ensures assessment of essential skills LEARNING 14 14 Page 7 7
2: Essential in CBME Structure Outcome Process (Competence) Time Entrustment 15 15 3: How People Become Experts Deliberate Practice Working on well defined tasks Informative feedback Repetition Self-reflection Motivation Endurance 16 16 Page 8 8
Coaching Requires Direct Observation 17 17 4: Supervision “. . . closer supervision leads to fewer errors, lower patient mortality, and improved quality of care. . ..” 18 18 Page 9 9
Assessing Does THE PATIENT 19 19 Observation and Safe Patient Care Safe, effective patient-centered care Appropriate level of supervision** **a function of attending competence in context Trainee performance* * a function of level of competence in context Kogan JR et al. Acad Med ; 2014;89:721-7 20 20 Page 10 10
Roadmap Review rationale for direct observation Identify barriers to frequent, high quality WBA Describe strategies to improve WBA 21 21 Lack of Direct Observation: Faculty Lack of time Lack of buy-in Fear of undermining learner-patient relationship Low self-efficacy Content/skills assessed Standards Feedback approach Diagnosing learner/offering action plan 22 22 Page 11 11
Lack of Observation: Learners Lack of faculty time Lack of trust, longitudinal relationships Anxiety provoking Artificial, “check-box” activity Threatens autonomy and efficiency Tension feedback vs high stakes assessment Berendonk C et al. Adv Health Sci Educ . 2013;18:559-71 Kogan JR et al. Med Educ. 2011;45:1048-60 Kogan JR et al. Med Educ. 2012;46:201-15 Govaerts MJB et al. Adv Health Sci Educ Theory Pract. 2011;16:151-65 Hatala R et al. Med Teach . 2008;30:414-19 Bing You RG et al. JAMA . 2009;302:1330-1 Watling C et al. Med Educ . 2012;46:593-60 23 23 Low Quality Assessments Poor accuracy Variability Focus of clinical performance Expectations about what is acceptable Rating errors Halo/“Horn” effect Leniency/stringency effect Cognitive bias 24 24 Page 12 12
Review Ratings of Video 25 25 Poor Inter-rater Reliability 26 26 Page 13 13
Frames of Reference 1 2 3 4 5 6 7 8 9 Unsatisfactory Satisfactory Superior ORDINAL Below At Expectation Exceeds Expectation Expectation NORMATIVE ???? ???? ???? GESTALT Kogan JR et al. Med Educ. 2011;45:1048-60 27 27 1 2 3 4 5 6 7 8 9 Not Close to What I Do What I Do What I Do (or better) SELF Missing evidence Most evidence All evidence based elements based elements based elements BEST PRACTICE Kogan JR et al. Med Educ. 2011;45:1048-60 28 28 Page 14 14
29 29 Assessors’ Own Clinical Skills Variable, sometimes deficient History taking Physical exam Counseling/shared decision making Patient centered communication Ramsey PG et al. JAMA 1993;269:1655-60 Paauw DS et al. JAMA 1995;274:1380-2 Vukanovick-Criley JM et al. Arch Intern Med. 2006;166:610-16 Braddock CH 3 rd et al. J Gen Intern Med. 1997;12(6):339-45 Kogan JR. et al. Acad Med. 2010;85(S10):S25-8 Levinson W. BMJ Qual Saf 2011;20:823-5 Frankel RM et al. Acad Med 2011;86:445-52 30 30 Page 15 15
Faculty Skills and Ratings of Learners Faculty with higher history and patient satisfaction performance scores provide more stringent ratings. Kogan JR. et al. Acad Med. 2010;85(10 Suppl):S25-8 31 31 High Level Inference Inferences are not recognized Inferences are rarely validated for accuracy Inferences can be wrong Govaerts MJB et al. Adv Health Sci Educ Theory Pract. 2011;16:151-65 Kogan JR et al. Med Educ. 2011;45:1048-60 32 32 Page 16 16
Contextual Factors Encounter complexity Resident characteristics Institutional culture Kogan JR et al. Med Educ. 2011;45:1048-60 Kogan JR et al. Med Educ. 2012;46:201-15 33 33 Roadmap Review rationale for direct observation Identify barriers to frequent, high quality WBA Describe strategies to improve WBA 34 34 Page 17 17
Strategies Getting faculty buy-in Finding time Minimizing interference Improving observation quality Agreeing on the standard Activating learners Maximizing the system 35 35 Strategies Getting faculty buy-in Finding time Minimizing interference Improving observation quality Agreeing on the standard Activating learners Maximizing the system 36 36 Page 18 18
Who Watched You? Being Observed •How did it feel? •Was it useful? Being the Observer •How did it feel? •Was it useful? 37 37 Highlighting Benefits of Observation 38 38 Page 19 19
Four Arguments for WBA Clinical skills matter but are variable Central in CMBE Essential for deliberate practice Necessary for effective supervision 39 39 Strategies Getting buy-in Finding time Minimizing interference Improving observation quality Agreeing on the standard Activating learners Maximizing the system 40 40 Page 20 20
Solution: Observation Snapshots What direct observation snapshots can you identify? *** GOAL: HELP THE LEARNER AND THE PATIENT *** 41 41 Observation Two and Three for Ones 2-for-1 (saves time)? 3-for-1 (saves time + helps patient)? 42 42 Page 21 21
Example Snapshots PHYSICAL PROCEDURES INTERVIEW COUNSELING EXAM Agenda Post-rounds Part of exam Consent setting Discharge instructions Pre-rounds Procedure Part of history Starting medication Post-check Pre-rounds Behavioral change Family meeting Code status Pre-rounds 43 43 CD/PD Helps Prioritize Skills 44 44 Page 22 22
Divide and Conquer 45 45 Strategies Getting buy-in Finding time Minimizing interference Improving observation quality Agreeing on the standard Activating learners Maximizing the system 46 46 Page 23 23
Minimize Interference Be a “fly on the wall” Avoid eye contact with patient Redirect eye contact Triangulation 47 47 Strategies Getting buy-in Finding time Minimizing interference Improving observation quality Agreeing on the standard Activating learners Maximizing the system 48 48 Page 24 24
Recommend
More recommend