■ ■ ■ ■ ■ ■
2019 019 summer er ser eries es Sally y Collier lier Chief Regulator
Evaluation uation of re refor orm m Jul ulie e Swan n Executive Director, General Qualifications Janet net Holloway lloway Associate Director, Standards for Design, Development & Evaluation of General Qualifications Be Beth h Bl Black k Director, Research and Analysis
■ ■ ■
Overall, for GCSE middle and upper band students, it is more challenging and more interesting (GCSE PE This was a tough but great teacher) paper. It was accessible for all but had plenty of scope to There were some show what they could do clever discriminating (GCSE history teacher) questions that identified the students ■ Generally satisfied – outcomes mainly as expected who really understood the science and the ■ Command words and mark scheme expectations task ■ Comparison with sample assessment materials (GCSE chemistry ■ Timing teacher)
It feels like a pressure cooker for getting grades out in terms of measuring progress ■ Tier entry and accountability and the accountability measures ■ Aspirational target grades (GCSE science teacher) ■ Perceptions and understanding ■ Progression to A level (entry requirements) ■ Mixed tier Exam boards and Ofqual could do more to inform school leadership teams about the overlapping grades – the messaging around grade 5 students being better suited to FT needs to be louder/stronger (GCSE science teacher)
Questions that assess practical skills are much better than controlled assessment (GCSE science teacher) The mathematical skills were shoehorned in and not necessarily driven by So little of the fieldwork done was the subject – artificially actually assessed mathematical (GCSE geography teacher) (GCSE science teacher)
■ ■ ■ ■
Evaluatio uation n & Res esea earch rch Beth th Black Di Direct ector or, , Research arch & Anal alysi ysis 13
■ □ □ □ ■ □ □ □ ■ 14
Ev Evaluating uating the impact ct of A l f A level el qua ualification ification refo form rm on n science ence practical ctical skills ills
The he ass ssessm essment ent of p f pract ctical ical work k in new A l A levels els Practical work directly assessed by teachers throughout the A level course Assessment of a minimum of 12 practical assignments (per subject) Performance assessed against ‘Common Practical Assessment Criteria’ (CPAC) Students receive a separate result (Pass/Unclassified) Min. 15% of marks in written exams allocated to indirect assessment of practical skills 16
The he ass ssessm essment ent of p f pract ctical ical work k in new A l A levels els 17
Ofqual’s research programme • Study 1: Teacher interviews – Perspectives on A level reform after one year • Study 2: Pre and Post reform evaluation of practical ability – A comparison of science practical skills in pre and post reform cohorts of undergraduate students • Study 3: Valid discrimination in practical skills assessment – An exploration of classification reliability when assessing the performance of practical skills • Study 4: Technical functioning of assessment – An analysis of A level examination items that assess science practical skills 18
Ofqual’s research programme for A level science pr pract ctic icals als • Study 1: Teacher interviews – Perspectives on A level reform after one year • Study 2: Pre and Post reform evaluation of practical ability – A comparison of science practical skills in pre and post reform cohorts of undergraduate students • Study 3: Valid discrimination in practical skills assessment – An exploration of classification reliability when assessing the performance of practical skills • Study 4: Technical functioning of assessment – An analysis of A level examination items that assess science practical skills 19
St Study dy 2: 2: Rese searc arch h with th univer versities sities Research question: What impact have changes to A level science had on the practical skills obtained by students? Method: Quasi-experiment which compares the practical skills of two groups: a. New undergraduates who studied pre-reform A-level science (2016) b. New undergraduates who studied post-reform A-level science (2017, 2018) • Data collected at universities • Practical Skills Measure (PSM) • Biology, Chemistry and Physics 20
2: The PSM carousel St Study dy 2: ‘Carousel’ of practical tasks (15 mins for each) Observed and assessed by university lecturer or PhD student Focus is on direct assessment of practical skills Participants are ‘marked’ against binary assessment criteria Questionnaire about A-level attainment, level of confidence, and experience with practical work Subject Unis 2016 2017 2018 Biology 6 138 298 278 Chemistry 4 109 185 202 Physics 5 293 225 131 TOTAL 15 540 708 611 21
St Study dy 2: 2: Bi Biology ogy PSM SM st stati tions ons Task 3 : Determining concentration of Task 4 : Aseptic technique - streaking an unknown from a standard curve plates with mock culture 22
St Study dy 2: 2: Overall all pe perfo formance rmance by ta task sk - Phy hysics sics Tasks 1) Use of an oscilloscope to measure the EMF of a battery 2) Micrometre and Vernier caliper 3) Measuring resistance with a voltmeter and an ammeter 4) Preparation of a circuit 5) Measurement and timing with a pendulum 6) Use of a signal generator and an oscilloscope 23
St Study dy 2: 2: Overall all pe perfo formance rmance by ta task sk - Ch Chemistry istry Tasks 1) Setting up a burette 2) Thin layer chromatography 3) Setting up a reflux and distillation 4) Making up a standard solution 5) Iodine clock (kinetics) 24
St Study dy 2: 2: Overall all pe perfo formance rmance by ta task sk - Bi Biolo logy gy Tasks 1) Making a standard solution and 10 fold dilution 2) Using a compound high power microscope 3) Determining concentration of unknown from standard curve 4) Aseptic technique - streaking plates with mock culture 5) Using an eyepiece graticule 6) Field survey skills 25
■ Reasons to be optimistic: □ No evidence for a decline in practical skills □ Some tentative evidence for an increase in skills in biology in 2017 & 2018 □ Self-report evidence of an increase in frequency of practical work ■ Caveats: □ There are limitations to the sample and methodology □ Reformed qualifications – 4 th cohort starts in September… things may still ‘bed in’. 26
Evaluatio uation n of modu dular lar ver ersus s linea ear 27
28 Multi-stranded ■ Literature review of the impact of structure on teaching and learning ■ Account of the policy history across qualifications ■ Qualitative research talking to teachers before and after teaching the new linear GCSEs ■ Quantitative analysis of the impact of structure on results ■ Impact on ability to set and maintain standards ■ Economic impact – on schools and colleges, exam boards and the wider system
29
30 Weaker students Linear demands Linear overwhelmed by long term gives more linear content retention time to develop Linear Motivation from supports success in early Stress from deep modules learning regular modules Linear level Pros & Cons playing field Students switched on by Modular exams regular logistically modules burdensome Modular flexible in meeting needs Demotivatio e.g. SEND Linear n from resit stressful availability Linear allows more teaching time
31 Subject Coefficient Standard error Effect size English -0.001 0.002 0.001 Mathematics -0.034 0.002 0.032 Science 0.001 0.003 0.001 Separate 0.087 0.003 0.091 ■ ■ ■ ■
GCSE E mathem hematics atics and pr d probl oblem em solving ing 32
■ ■ □ 33
■ ■ 34
■ ■ ■ □ □ □ □ 35
■ 2017 assessments Sample assessments ■ ■ 36
A small decrease (2-3 %) in marking consistency with increase problem solving content. Have paid a tiny bit in reliability, in order to increase validity and value of the qualification. 37
■ □ □ □ ■ 38
Qu Ques estion tions? s?
Summer er award rding ing Ca Cath h Jadha hav Director, Standards and Comparability 41
A level entries up in: Biology, chemistry, physics, psychology, sociology Also, computing, politics and Spanish Down in: English subjects and maths
■ ■ ■ ■
■ ■ ■ ■
■ ■ □ □ ■ ■
Combined science tier split Foundation tier: 63% (55% in 2018) Higher tier: 37% (45% in 2018) Largest % increases in entries 1. Spanish 2. Computer science 3. History 4. Combined science 5. French
■ ■ ■ ■ ■ ■
■ ■ □ □ □ ■ ■ ■
■ ■ ■ □ □ □ ■
■ ■ ■ ■ ■
■ ■ ■
• Centre variability in results • Results by county • Grade distributions (9 to 1) Updated with 2019 on results days Analytics.ofqual.gov.uk
Recommend
More recommend
