VT VTQ Re Research: arch: Ge Getti tting ng Te Technical - PowerPoint PPT Presentation

VT VTQ Re Research: arch: Ge Getti tting ng Te Technical hnical Paul ul E. Newto ton AO O Foru rum, m, London, don, 14 May 2019

Technical hnical research earch into to GQ- like tests and exams be like… ■ Using Hierarchical Logistic Regression to Study DIF and DIF Variance in Multilevel Data ■ A Comparison of Strategies for Smoothing Parameter Selection for Mixed-Format Tests Under the Random Groups Design ■ Calculating Conditional Reliability for Dynamic Measurement Model Capacity Estimates

Technical hnical research earch into to VTQ- like assessments be like… Credit: Bill Applin

But isn’t VTQ assessment inherently non -te technical? hnical? The assumption [underpinning NVQs] has always been What I am proposing is that we shou ould ld just t for orge get t that asse sessment ssment will be unpr probl oblematic ematic because ause it t simpl ply y relia iability bility alto toge gether, ther, and d con oncentrate ntrate on on valid lidity ty, which is invo volves lves com omparing ring behavio aviour ur with th th the tr transparent sparent ultimately all that matters. ‘benchmark’ of the performance criteria . (Jessup, 1991, p.191) (Wolf, 1995, p.24)

But isn’t VTQ -li like asse sessme ssment nt inhere erently ntly non on-te technica chnical? l? The assumption [underpinning NVQs] has always been that asse sessment ssment will be unpr probl oblematic ematic because ause it t simpl ply y invo volves lves com omparing ring behavio aviour ur with th th the tr transparent sparent ‘benchmark’ of the performance criteria . Unfo fortu rtunately nately, in practice tice th this tu turns ns ou out n t not ot to to b be th the case. e. (Wolf, 1995, p.24) ■ … even NVQ -like Competence-Based Qualifications are far from unproblematic, and require a technical eye; ■ … and the more ‘GQ - like’ a Competence -Based Qualification becomes, the more of a technical eye it’s likely to require.

Becoming ‘GQ - like’? ■ “Professor Wolf’s report is very clear that assessment methods for many vocational qualifications need to be strengthened […] Therefore, on only th thos ose quali lificatio fications ns th that t prov ovid ide evidence dence of of substantial stantial amou ount nt of of exte ternal rnal asse sessme ssment nt , together with synoptic assessment […] will be counted in the tables.” ■ “So we will on only includ lude e th thos ose quali lificatio fications ns th that t are grad aded ed – as opposed to a pass/fail – in the tables in the future. Qualifications may have a pass, merit, distinction structure or a more detailed scale.”

https:/ ://w /www.gov.uk ww.gov.uk/g /govern overnme ment/ nt/pu public licatio ions/gr ns/grading ding-voca vocation tional al-and nd-techn echnica ical-qual qualif ificat icatio ions

Level el 1 NVQ Certificat tificate e in Hos ospital italit ity y Servic vices es … comparing behaviour with the transparent ‘benchmark’ of the performance criteria?

Level el 2 & 3 D Diploma oma in Skil ills ls for or Busine siness ss Level el 2 (unit Level el 3 (unit nit 10) nit 10) LO1 Understand the reasons for change in LO1 Understand change in business business AC1.1 State why it is important for a business AC1.1 Explain why it is important for a to change business to change AC1.2 State the risks associated with a AC1.2 Analyse the positive and negative business changing too quickly effects of change on a selected AC1.3 State the risks associated with a AC1.3 Compare the risks of slow against business changing too slowly rapid change within a business AC1.4 Compare the benefits of slow against rapid change within a business

Unit 10 Pass Merit Distinction The candidate will The candidate will [No D for this AC] L2 state state why it is important for a why it is important for a AC1.1 business to change business to change, demonstrating critical understanding The candidate will The candidate will The candidate will give L3 explain explain a sophisticated in detail explanation of AC1.1 why it is important for a why it is important for a why it is important for a business to change business to change business to change

All sor orts ts of of te technical nical question tions s relat ated ed to to grad ading ing ■ Standardisation ■ Comparability ■ Grading and levelling ■ Weighting ■ Burden and backwash ■ Transparency

We know ow how ow hard rd chall llenges enges like e sta tanda ndardisa rdisati tion/ on/co compa mparabi rabili lity ty can be ■ Because we’ve done a lot of technical research into issues like these, for GQ -like qualifications.

Grad ade infla flation tion at t A level el? Cumulative percentage of A level students awarded grade E (or higher) (for All Boards, Summer Awards, All Modes, by Syllabus Group) 100 95 90 85 Biology Latin 80 Physics 75 Sociology 70 65 60 1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009

Grad ade infla flation tion at t A level el? Cumulative % candidates awarded A at A level JCQ (June) Data, All UK Candidates, All Subjects 30 28 26 24 22 20 2002 2003 2004 2005 2006 2007 2008 2009

No l o lon onger er any y evidence dence of of grad ade e infla flation tion at t A level Cumulative % candidates awarded A at A level JCQ (June) Data, All UK Candidates, All Subjects 30 28 26 24 22 20 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018

https://w s://www. w.go gov. v.uk uk/go gove vernm nment/ ent/pu publ blic icat ations ions/stre strengt ngthe hening ing-vocat vocational ional-and and-tec technical hnical-qual qualifica ification tions Ben Cuff, Nadir Zanini, Beth Black

A Level BTEC (Subsidiary Diploma)

We know ow how ow hard rd chall llenges enges like e sta tanda ndardisa rdisati tion/ on/co compa mparabi rabili lity ty can be ■ But is it particularly challenging to maintain a firm grip on standards within Competence-Based Qualifications, like ‘older style’ BTECs?

Becoming ‘GQ - like’? ■ “Professor Wolf’s report is very clear that assessment methods for many vocational qualifications need to be strengthened […] This helps ps to to ensure ure that vocational qualifications offer a com omparabl rable e level l of of chall llenge enge to academic qualifications and are seen to do so. External assessment also prov ovid ides es an addit ditio ional nal check k th that t sta tandards ndards are con onsiste istent nt across centres.”

We know ow what t a goo ood GQ te test t loo ooks like ke ■ But what does a good VTQ test look like?

What t makes s a goo ood te test? t? 1. Questions (and test overall) should: a) be of an appropriate level of difficulty b) differentiate between learners, but (only) on the basis of their proficiency 2. Test overall should: a) deliver reliable results b) embody standards that are comparable with comparable tests

https:/ ://w /www.gov.uk ww.gov.uk/g /govern overnme ment/ nt/pu public licatio ions/a ns/asse sess ssme ment nt-funct unctio ionin ning-of of-exte xterna nal-asse sessmen sments ts Beth Black, Qingping He, Stephen Holmes, Caroline Morin

VTQs on the ‘approved list’ for DfE’s 16 -19 perfo formance rmance ta table les 2016 serie ries 2017 serie ries ■ 49 tests (27 qualifications) ■ 20 tests (15 qualifications) ■ mainly L1 and L2 ■ L3 only (Applied Generals/Tech Levels) ■ health & social care, carpentry, hospitality, digital ■ applied science, business, digital media, engineering, media, applied science, mathematics health & social care, IT/computing, sport

Ite tem funct ctioning ioning sta tati tistics stics – com ompute ted d separate rately ly for or each h question tion ■ Facility indices □ how easy/hard is the question? ■ Discrimination indices □ do candidates who tend to perform poorly/well on the question also tend to perform poorly/well on the test overall?

Test t funct ctioning ioning sta tatistics tistics – com ompute ted d for or th the te test t ov overal rall ■ Mean of marks □ how easy/hard is the test? ■ Standard deviation of marks □ how widely spread are the marks (across the mark range)? ■ Reliability coefficient □ an estimate of the degree to which results are likely to be replicable

We need d to to m make e certa tain in assump umption tions ■ Each test is intended to provide an ov overal rall esti timat mate of proficiency □ it is not intended to certify the attainment of a specified set of AC for the unit ■ Each test is intended to differe fferentia ntiate te between candidates □ between gradations of proficiency (pass, merit, distinction) ■ All candidates were adequately pr prepa pared red for their tests □ they were enrolled on an appropriate course (at the right level) □ they were taught the subject content appropriately □ they were given appropriate experience of test-taking

2016 te tests ts – Facili cility ty indi dice ces