Are the GMC’s Tests of Competence fair to long standing doctors? L Mehdizadeh, A Sturrock, J Dacre Presented by Dr Leila Mehdizadeh Research Associate UCL Medical School l.mehdizadeh@ucl.ac.uk
Academic Centre for Medical Education • Research department of UCL Medical School • Remit for educational research • Contract with GMC FtP until 2019 • Design and delivery of the Tests of Competence (ToC)
GMC performance assessment On site • Peer led work based assessment Off site • Test of competence - Knowledge test (SBA’s) - Practical skills test (OSCE)
Tests of competence • The ToCs are not pass/fail examinations • The results will be considered with all other evidence when coming to a judgement about the doctor’s FtP • The evidence may however be subject to challenge by defence lawyers
Study participants • Doctors under investigation • Volunteer doctors - pilot the ToC material - reference group for comparison
Study rationale • Average age of doctor undergoing performance assessment was 55 • High proportion of doctors under investigation known to be overseas trained • SBA and OSCE are newer test formats that some doctors may be unfamiliar with
Qualification year and test preparation • Evidence that increase in age and time elapsed from qualification year is associated with decline in clinical performance (Chaudhry et al 2005; Duclose et al 2012; Caulford 1994). • Test preparation and test familiarity is linked to improved clinical performance (Goldberg and Pedulla 2002, Bangert-Drowns, & Kulik, 1984).
Research questions Does year of qualification predict ToC performance • in doctors undergoing FtP investigation? • in doctors with no known concerns? • when controlling for gender, ethnicity and region of PMQ in both groups of doctors?
Methods Retrospective cohort design Compared 95 GPs under FtP investigation vs 376 volunteer doctors Outcomes - knowledge test score (total 120) - OSCE overall score (total 480)
FtP group Volunteer group Total on GP register Results: demographics (n = 95) (n = 376) (n = 63,778) Gender 79 (83%) 169 (45%) 32, 719 (51%) Male Female 16 (17%) 207 (55%) 31, 059 (49%) PMQ region 34 (36%) 324 (86%) 49, 198 (77%) UK EEA 7 (7%) 4 (1%) 3, 998 (6%) International 54 (57%) 48 (13%) 10, 582 (17%) Ethnicity 27 (28%) 220 (59%) White Black 17 (18%) 16 (4%) 39 (41%) 112 (30%) Asian 0 8 (2%) Mixed Other 12 (13%) 18 (5%) 0 2 (<1%) Not stated Year of qualification Before 1993 (early) 85 (89%) 69 (18%) 10 (11%) 146 (39%) 1993-2005 0 161 (43%) 2006-2011 (recent)
Knowledge test performance: volunteer group
Doctors under investigation
OSCE performance: volunteer group
Doctors under investigation
Multivariable linear regression Summary of effect of qualification year when controlling for gender, ethnicity Qualification year was and qualification region more strongly associated with ToC Doctors under Volunteer doctors performance in FtP investigation doctors than volunteers Performed worse Performed better than FtPs than volunteers on on all outcomes all outcomes Earlier graduates Recent graduates performed performed worse worse than earlier graduates than more recent on KT, but on OSCE those graduates on KT who qualified after 2005 and OSCE overall performed best. Doctors under investigation who qualified prior to 1996 performed significantly worse on KT and OSCE overall.
Discussion • Relationship between qualification year and ToC performance differs between two groups • If ToC format was unfair to older doctors we would expect to see similar pattern in volunteers • Why does performance of FtP doctors decline? • Why do recently qualified volunteers do worse in KT and better in OSCE?
Strengths and limitations Large sample, few studies explore effect of qualification year on high stakes exam Demographics differ between two groups Most FtP qualified prior to 1992 vs volunteers from 2006+ Volunteers that graduated before 1996 performed better on all outcomes Data only for general practice
Does qualification year predict ToC performance? • Yes in doctors undergoing investigation • Less so in volunteer doctors
Take home messages ToC are fair irrespective of when doctors qualified Need to extend study with: -more doctors under investigation who are recent graduates -more volunteer doctors who are early graduates - data from range of clinical specialties - control for doctors’ age and years in practice
Thank you for your attention l.mehdizadeh@ucl.ac.uk
Recommend
More recommend
