RESIDENCY-IN-TRAINING EVALUATION (RITE) ANNUAL WRITTEN EXAMINATION - PowerPoint PPT Presentation

RESIDENCY-IN-TRAINING EVALUATION (RITE) ANNUAL WRITTEN EXAMINATION & INTERNAL MEDICINE ADMISSIONS TEST(IMAT) 2013 Lenora Fernandez, MD, FPCP

OVERVIEW on RITE • RITE/REE held yearly 1987-2003 then stopped when validity & reliability in doubt • Voluntary on-line format 2009-2011: very few takers • Annual written exam format resumed 2013 = OCTOBER 13, 2013, Sunday

OBJECTIVES of RITE General Objective: • To determine the cognitive skills of internal medicine residents in all PCP-accredited training programs. Specific Objectives: – To compare the cognitive skills of IM residents among all accredited internal medicine training programs in the Philippines. – To compare the cognitive skills between 2 nd and 3 rd year IM residents within the institution and all other institutions. – To determine the areas of strengths and weaknesses of the examinees and serve as guide in maintaining and improving their training programs. – To provide a tool in setting standards for accreditation of IM training programs to the Committee on Accreditation in (future) – To determine if this examination can predict the performance of the residents in the PSBIM.

OVERVIEW on RITE • Multidisciplinary committee Cardiology Allergology-Immunology Pulmonary Medicine Rheumatology Endocrinology Medical Oncology Gastroenterology Neurology Nephrology Dermatology Infectious Diseases Ethics Hematology • Tried to simulate the PSBIM • 200 items: 175 MCQs, 25 Modified True or False Recall - 30% Based on PCP Glossary of Top Comprehension – 30% Diseases Analysis – 40% • Chapters hosted and conducted the RITE

23 RITE TEST SITES Fatima College of Medicine, Quezon City Valenzuela City (CAMANAVA) Manila City Laoag City Marikina City Dagupan City Makati City (MATAPAT) Baguio City Las Pinas City (PAMUNLAS) Tuguegarao City Pasay City Legaspi City Pasig City (PASJMAN) Lipa City Angeles City, Pampanga Tacloban City Cebu City Davao City Tagbilaran City Cagayan de Oro City Iloilo City Zamboanga City Bacolod City

RITE DEMOGRAPHICS Total number of examinees 1264, 94 training institutions Year Level 2 examinees N = 476 Mean score = 102.38/200 Year Level 3 and 4 examinees (no.) 469 Mean score = 106.1/200 Post-residency examinees (no.) 319 Mean score = 94.7 Set minimum passing level*: Year Level 2 86 out of 200 Year Level 3, 4 and post-residency level 107 out of 200 RITE exam analysis: Range: 65.2-153.8 Mean: 101.8 Median: 100.8 Mode: 108.2 Reliability index: 0.83 (vs 0.63-0.78 in previous) * MPL pre-determined: YL 2 – 10% lower than YL3/4 (71% of 200 items); MPL lowered by 1 SD after results collated

RITE TEST INDICES Difficulty index: Discrimination Index: 10% very easy < 1% very poor 25% easy 50% poor 15% moderate 30% good 30% difficult 20% excellent 20% very difficult

MEAN RITE SCORES IN 3 GROUPS p < 0.001 p < 0.001 106.7 + 15.3 110 102.4 + 14.6 105 Mean RITE Scores (over 200 items) 100 94.7 + 11.67 95 90 85 80 2nd year 3-4th year Postgraduate N = 476 469 319

PERFORMANCE IN DIFFERENT TOPICS OF THE 3 GROUPS YL2 YL3-4 PGRAD % mean score in topic 80.00% 60.00% 40.00% 20.00% 0.00%

TOP 25 TRAINING INSTITUTIONS IN PERCENTILE RANK, RITE UP-PGH National Kidney & Transplant Institute University of Santo Tomas Premiere Medical Center Cardinal Santos Medical Center Zamboanga City Medical Center Ospital ng Maynila Medical Center FEU-NRMF Medical Center Makati Medical Center Davao Doctors Hospital Western Mindanao Medical Center St Lukes Medical Center UERM Memorial Medical Center SLU-Hospital of the Sacred Heart Chong Hua Hospital West Visayas State University Medical Center University of Perpetual Help Dr. Jose G. Tamayo Medical Center-Binan, Laguna Southern Philippines Medical Center Divine Word Hospital Chinese General Hospital and Medical Center St. Paul's Hospital AFP Medical Center Manila Adventist Medical Center Mary Mediatrix Medical Center Dr. Pablo O. Torre Memorial Hospital

YEAR LEVEL 3 RITE - PSBIM 114.41 + 12.84 94.45 + 9.38 98.79 + 13.64 120 100 Mean RITE Scores (over 200 items) 80 p < 0.001 60 40 20 0 Passed PSBIM Failed PSBIM Did not take PSBIM N = 232 51 186

POSTGRADUATE EXAMINEES: RITE - PSBIM 102.7 + 10.4 104 p < 0.001 102 100 Mean RITE Scores (over 200 items) 98 93.5 + 10.16 96 90.75 + 10.4 94 92 90 88 86 84 Passed PSBIM Failed PSBIM Did not take PSBIM N = 79 116 124

FEEDBACK OF EXAMINERS/PROCTORS • Clear instructions? YES- 100% • Enough time to prepare for organization of RITE? YES – 96.1% • Conflicts in the instructions given? NONE 78.7% Do you agree to the involvement of PCP chapters in the conduct of • RITE? YES – 99% Will you be willing to serve again as proctor or organizer for the • RITE in the coming years? YES- 98% Further comments: • – Role of RITE in regular evaluation of residents? – Notify earlier, orientation meeting, minimize conflicts in instructions – Separate exam for 2 nd and 3 rd years – Include first years – Chairmen to encourage post-residency members who want to take PSBIM – Compensation to staff?

FEEDBACK OF EXAMINEES • Was RITE given at the right time of your training? YES 86.7% • For residents-in-training, test questions relevant and appropriate to your level of training? YES 98.3% For post-graduate examinees, test questions serve as a “good review” for • your coming PSBIM exam? YES 98.4% • Enough time allotted to answer all questions? YES 96.2% • Enough time to review for the exam ? YES 55.33% • Details provided beforehand sufficient? YES 87.6% • Venue conducive to answering exam well? YES 91.2% Proctors helpful in clarifying your questions during the exam? YES 99.7% • • Will you recommend that RITE be regularly conducted on an annual basis? YES 95.1% • Would you prefer RITE to be conducted on-line? YES 37.4%

FEEDBACK OF EXAMINEES • Must include first years, more often • Venue issues (temp, announcement, chairs/desk, comfort room, crowding) • Questions (different from PSBIM, relevance) • No questions allowed • Coverage/blueprint not announced earlier • Must be paired with a post-exam review

CONCLUSIONS ON RITE ANNUAL WRITTEN EXAM: • The RITE 2013 met most of specific objectives it aimed for: – compare the cognitive skills of IM residents among all IM training programs. – DONE – compare cognitive skills between 2 nd and 3 rd year IM residents within the institution and all other institutions. - DONE – determine the areas of strengths and weaknesses of the examinees and serve as guide in maintaining and improving their training programs. – DONE – determine if this examination can predict the performance of the residents in the PSBIM.  Based on the results, it seems it can predict the performance in the PSBIM. • Reliability index high at 82.9% + opinion of majority among the examiners and examinees to continue with RITE. • The RITE is recommended to be continued.

RECOMMENDATIONS • Reformulation of RITE Committee examiner composition . More training in test construction. • Next date: October 12, 2014, Sunday . • • More test sites nearer to examinees to be manned by Chapters. • Better and earlier coordination with the PCP chapters. • Test blueprint given today: – ALL MCQ’s – 250 items – include questions on research and principles on critical appraisal of literature – Integrative questions across different topics;subdiv to ambulatory, emergency, or critical care dimensions. Scantron machine already available at PCP • • More efficient system of dissemination and recording/collating • Data to be continually analyzed • Discussion on bearing on internal evaluation of resident will be opened?

INTERNAL MEDICINE ADMISSIONS TEST Objectives of IMAT: • Provide a uniform assessment standard for entrance into IM training programs • Create a profile of aspiring entrants • Determine baseline knowledge of individual aspiring entrants in comparison to the general pool of aspirants through normative statistical analysis.

OVERVIEW & CONDUCT • 100-item MCQ written exam • Questions to cover only pathophysiologic basis of disease, general diagnostic and therapeutic principles of general disease categories (cardio, pulmo, endo, IDS, nephro, etc.) • No MPL set • Given to TI’s with answer key for immediate utilization in their screening process; can administer at any time

CORRELATION OF IMAT WITH MEDICAL BOARD EXAM RATING 90 N = 266 evaluable results Overall Medical Boards Rating (%) 85 80 75 20 40 60 80 100 IMAT Score (raw over 100 items) Pearson’s correlation coefficient = 0.687

CHALLENGES: • Lack of security of the exam: – Leakages unavoidable since administered at different dates • Provision for applicants to different institutions • Relevance in screening process? – What is/are main factor/s in accepting new residents into training program?

RECOMMENDATION: • CONSENSUS decision among Regents and Training Institutions to STANDARDIZE SCREENING PROCESS for IM training applicants with a written standardized examination as just ONE of the screening tools – Uniformity in time of screening can then be ensured – Security and confidentiality then of IMAT can be ensured