Removing barriers: Financial transparency, peer effects and prior attainment
Team: � Simon Burgess, Julia Carey at EFM, University of Bristol � Catherine Dilnot, Lindsey Macmillan at UCL/IOE � Raj Chande, Michael Sanders at Behavioural Insights Team � Also � Abbi Stoneman, Christine Spencer and Maggi Walton, Project administrators � Kate Guest from Teach First ran the mentor training � Amy Butterworth, Deputy Head of Admissions at UoB helped with understanding data and provided admissions data � All the student mentors. UoB WP Workshop, Oct. 2016 2
This talk � Issue � ‘Model’ and Interventions � Aims � Operations � Analysis: � Methodology � Data � Preliminary results � Timetable � Conclusions UoB WP Workshop, Oct. 2016 3
Issue � Despite huge rise in numbers attending university, the rate of return is still high for most students. � How can we help more students from disadvantaged backgrounds attend the best universities? � It may all be about bursaries … But recent research suggests that other factors may matter as much. UoB WP Workshop, Oct. 2016 4
‘Model’ and Interventions � We believe there are other important barriers to access: � Not understanding the true financial costs and benefits of going to a Russell Group university � Not believing that a Russell Group uni is “for people like me” � Not having the very top grades to get in � Our interventions address these � We train UoB students to go into local schools to deliver: � ‘Inspire’ talks � Additional small-scale tutoring UoB WP Workshop, Oct. 2016 5
Aims � The over-arching aim was to: � improve access to the University of Bristol … � … and universities more broadly … � … for disadvantaged children in local schools and colleges. � We will provide causal evidence on the efficacy of this approach using an RCT methodology. � We have written a Handbook on managing this scheme. UoB WP Workshop, Oct. 2016 6
Operations � Intervention from 2013/14 to 2015/16, finishing June 2016. � Recruit local schools to take part � ‘Local’ was broad including disadvantaged areas of Weston-super-Mare. � Included regular state secondary schools and Further Education colleges � Recruit UoB students to act as mentors: � Fresher’s Fair and Volunteer’s Fair, and later through email , undertaken each year � Work with Teach First to train the students for both roles � Teach First do this for free, undertaken each year. � Match mentors to schools, and organise the practicalities: � Funded the travel of the mentors but they were volunteers, so not paid. UoB WP Workshop, Oct. 2016 7
Analysis � Methodology � Data � Preliminary Results UoB WP Workshop, Oct. 2016 8
Methodology � RCT approach � School-year groups are randomised into Control, ‘Inspire’, ‘Tutoring’. � Control = “business as usual”. � ‘Inspire’ talks: � We treated whole school-year groups � School-year group level analysis � Tutoring � Five pupils selected by school (not random) for tutoring � Pupil-level analysis using matched pupils in Control schools UoB WP Workshop, Oct. 2016 9
Data � ‘Inspire’ talks: school-year-calendar year level data on applications from UCAS: � UoB � any RG � any HE � Mentoring: pupil level data from NPD: � GCSE grades � A level grades UoB WP Workshop, Oct. 2016 10
� Sample: 43 schools and colleges in the trial: � 7 are 16+ colleges only (so only appear in KS5 data) � 10 have no sixth form (so only appear in KS4 data) � 26 schools are all through, so appear in both UoB WP Workshop, Oct. 2016 11
Trial schools State schools and colleges: In the Trial All in Bristol, In England Gloucestershire and Somerset KS4 KS5 KS4 KS5 KS4 KS5 Number of students 6,857 7,594 23,971 16,776 593,435 395,457 Number of schools/colleges 36 33 174 98 3,775 2,392 % female 48.1 53.7 48.6 52.5 48.8 53.2 FSM (% of non-missing) 12.7 7.0 10.3 5.7 14.9 10.7 SEN (% of non-missing) 13.9 8.3 14.3 8.2 17.0 10.4 SSEN (% of non-missing) 2.2 0.9 3.4 0.7 3.8 1.1 EAL (% of non-missing) 7.5 6.3 4.7 4.9 12.7 15.1 Ethnicity: % Asian (of non-missing) 3.6 2.9 2.0 2.3 8.2 10.3 Ethnicity: % Black (of non-missing) 4.0 2.6 1.9 1.8 4.8 5.7 Ethnicity: % Mixed (of non-missing) 4.2 3.9 3.1 3.3 3.8 4.0 Ethnicity: % White (of non-missing) 87.5 89.2 91.8 91.0 80.6 76.9 Ethnicity: % AOEG/Chinese/Unclear (of 0.8 1.4 1.2 1.6 2.6 3.1 non-missing) GCSE mean capped points score 333 370 332 377 333 375 including equivalents % with 5 A*-C – GCSE and GNVQ 53.5 79.4 56.8 82.2 55.1 78.3 % with A-levels as main qualification N/A 56.9 N/A 61.4 N/A 57.5 % with NVQ as main qualification N/A 41.3 N/A 36.7 N/A 39.9 % achieving 3 or more A*-E grades in A- N/A 42.5 N/A 48.3 N/A 46.6 levels/applied A-levels Mean points from A-levels (across all N/A 415 N/A 464 N/A 445 students) Mean number of A-level entries per N/A 2.0 N/A 2.2 N/A 2.1 student Mean points per A-level entry N/A 199 N/A 205 N/A 202 UoB WP Workshop, Oct. 2016 12
Trial schools – extract … State schools and colleges: In the Trial All in Bristol, Glos & Somerset KS4 KS5 KS4 KS5 FSM (% of non-missing) 12.7 7.0 10.3 5.7 EAL (% of non-missing) 7.5 6.3 4.7 4.9 % with 5 A*-C – GCSE and GNVQ 53.5 79.4 56.8 82.2 % with A-levels as main N/A 56.9 N/A 61.4 qualification % with NVQ as main qualification N/A 41.3 N/A 36.7 % achieving 3 or more A*-E grades N/A 42.5 N/A 48.3 in A-levels/applied A-levels UoB WP Workshop, Oct. 2016 13
Preliminary results � Our research is ongoing and these are preliminary findings. � Our determination to give the actual implementation as long as possible to run, and the relative slowness of the process of accessing NPD data means that there is more to do. UoB WP Workshop, Oct. 2016 14
Results 1: Tutoring � We provided tutoring to 108 students from year 9 to 11 over the three years. NPD matched 106. � Only 29 appear in KS4 data that we have so far. � So – small numbers and very preliminary. � Control group from same LAs as our schools, matched using very local SES score, KS2 and gender. � Capped GCSE scores as a summary measure: UoB WP Workshop, Oct. 2016 15
Treated Controls CGSE Difference GCSE score score Unmatched 373 302 71 ATT 373 342 31 So about an 8% increase in GCSE points PRELIMINARY UoB WP Workshop, Oct. 2016 16
Results 2: ‘Inspire’ Talks � Outcome measure is UCAS “applications” to UoB, RG, All HE. � Analysis uses the randomisation plus a difference-in- difference approach � Include school fixed effects and application-year effects. UoB WP Workshop, Oct. 2016 17
All HE Applications (1) HE applications (total) b/se treat 14.948 (65.521) application_cycle=2013 0.000 (.) application_cycle=2014 50.441 (48.704) application_cycle=2015 -4.546 (51.549) application_cycle=2016 29.831 (52.507) Constant 801.487*** (34.819) N 95 (About 2% baseline; if each kid applies to 5 unis, that’s almost 3 more kids) UoB WP Workshop, Oct. 2016 18
Russell Group Applications (1) RG applications (total) b/se treat 0.528 (11.030) application_cycle=2013 0.000 (.) application_cycle=2014 21.462* (8.199) application_cycle=2015 4.872 (8.678) application_cycle=2016 21.933* (8.839) Constant 124.829*** (5.862) N 95 UoB WP Workshop, Oct. 2016 19
UoB Applications (1) UoB applications (total) b/se treat 1.440 (2.411) Application cycle=2013 0.679 (1.829) Application cycle=2014 1.804 (1.829) Application cycle=2015 0.000 (.) Application cycle=2016 3.363+ (1.772) Constant 22.439*** (1.358) N 94 UoB WP Workshop, Oct. 2016 20
UoB Apps: Longer run of data UoB WP Workshop, Oct. 2016 21
UoB Apps: “before” and “after” UoB WP Workshop, Oct. 2016 22
Commentary 1 � � Preliminary results � � No statistical significance, but that might change � Positive results, very worthwhile but not huge: � ‘Inspire’ Talks on � All HE applications � ‘Local RG’ university = University of Bristol � Tutoring on � GCSE points UoB WP Workshop, Oct. 2016 23
Commentary 2 � Policy Lessons? � To the extent that UoB WP own practices are somewhat similar to ours, suggests that they may be effective too. � But two caveats: � Control condition was “business as usual” so that includes the activities of the University’s own WP team. Our intervention raised applications above and beyond their activities, suggesting pupils don’t see them as the same. � No power to isolate the impact of the different components of our intervention. So cannot know whether the most effective components are or are not part of UoB existing practices. UoB WP Workshop, Oct. 2016 24
Timetable � Sharpening results on Inspire Talks � Now through December 2016. � Revising results with the 2016/17 Admissions data, Spring 2017. � Undertaking pupil level analysis on mentoring � Spring 2017, using the final release pupil level data for 2016 KS4, available next April. � Dissemination � Late Spring 2017 � We have good links with: the Russell Group, individual leading universities, the Cabinet Office, the Department for Education, the Social Mobility Commission UoB WP Workshop, Oct. 2016 25
Recommend
More recommend
