Improving the value of policy evaluation Nicholas Mays Professor of - PowerPoint PPT Presentation

Improving the value of policy evaluation Nicholas Mays Professor of Health Policy Department of Health Services Research and Professor of Health Policy, Department of Health Services Research and Policy Treasury guest lecture, 29 January 2013 Treasury guest lecture, 29 January 2013 Improving health worldw ide p g w w w.lshtm.ac.uk

Outline Outline Focus of presentation Focus of presentation • Meaning of ‘value’ of policy evaluation • Perceived problems with evaluation in public policy p p p y • Current proposals for increasing the ‘value’ of policy evaluation • – Supply – Demand – Interactive Critique of proposals • Conclusions and questions for discussion •

Scope Scope External evaluation (mainly) ( ) • Ex ante and ex post evaluation • E Evaluation involving new data collection and secondary analysis l ti i l i d t ll ti d d l i • UK, NZ (mainly) • National level social policy National level social policy • • Not discussing technical quality of evaluations or dissemination and • knowledge transfer g Partially drawing on recent experience of trying to improve the • ‘value’ of policy evaluation through directing a government ‐ funded research unit in England (www.piru.ac.uk/) h l d ( k/)

‘Value’ of ‘evaluation’ Value of evaluation • ‘Evaluation’: – ‘systematic assessment of the operation and/or the outcomes of a programme or policy’ (Weiss, 1998, p4) • Currently thinking of ‘value’ as: – ‘better potential to use’ • As much a result of policy processes, norms and p y p , expectations as a feature of evaluation itself

Perceived problems with policy evaluation Lack of understanding of the potential benefits of evaluation Lack of understanding of the potential benefits of evaluation • • even denigration of evaluation in some quarters as too late, irrelevant, unreliable, – incomprehensible, embarrassing ... Lack of commissioning of, consideration for the needs of, and use of, • evaluations ex ante and ex post Misuse or inappropriate use of evaluations • lack of research/statistical skills – Sectors other than health lagging behind in sophistication, availability and use S t th th h lth l i b hi d i hi ti ti il bilit d • of evaluation all sectors should be aspiring to be ‘evidence ‐ informed’ if not ‘evidence ‐ based’ policy – Typically the concern of researchers rather than decision makers though Typically the concern of researchers rather than decision makers though • periodic waves of interest from the latter 1960s, late 1990s and currently in UK/England (associated with wider civil service critique and – reform plans from Government) series of reports/initiatives in NZ (Review of the Centre 2001; MfO 2003; SPEaR; Better Public series of reports/initiatives in NZ (Review of the Centre, 2001; MfO, 2003; SPEaR; Better Public – – Services Advisory Group, 2011), though more about outcomes & targets across government current focus on evaluation justified by a context of financial stringency (Obama, UK Alliance for – Useful Evidence, French Experimentation Fund for Youth)

One topical call for the ‘good use’ of evidence (and one of the more sophisticated!) ‘It is exceptionally rare for scientific evidence to mandate a single solution to a policy problem; rather it informs the range of solutions that might be feasible and predicts what the outcome of each is that might be feasible and predicts what the outcome of each is most likely to be. (p54) .... Ministers aren’t obliged to make every decision according to the evidence presented to them by scientists and nothing else. They should, however, ensure that they do take d thi l Th h ld h th t th d t k scientific advice on questions to which it is most pertinent ... Above all, politicians and civil servants should not be allowed to get away , p g y with laying claim to evidence ‐ based policy when decisions have actually been taken by other means.’ (p56) Henderson M. (2012) The geek manifesto –why science matters. d ( ) h k f h

Recent reports arguing for change abound

Supply side proposals (research community) Supply side proposals (research community) More ex ante evaluation • – More modelling and ex ante assessment (answering today’s problems, today) – Greater willingness to use ‘best possible’ rather than ‘definitive’ evidence – Greater willingness to engage with policy decision makers and advise g g g p y More agile ex post evaluation • – e.g. more use of large, long ‐ term observational administrative databases More attention to setting out the ‘logic’ underlying policies More attention to setting out the logic underlying policies • – to assist with evaluation design and explaining how and why policies might ‘work’ – working with policy makers, to help clarify the intervention, population, intended outcomes and processes, and data needed for improved evaluation outcomes and processes, and data needed for improved evaluation Change to reward and recognition systems in universities • – more weight to ‘impact’; less to grants, original research and peer reviewed outputs

Demand side proposals: largely ex ante Demand side proposals: largely ex ante Raise the ‘absorptive capacity’ of policy organisations by giving them • – more permeable boundaries – stronger external networks (e.g. with research community) – stronger ‘organisational memory’ – incentives on individual policy officials (e.g. that they use evaluation in their work) i ti i di id l li ffi i l ( th t th l ti i th i k) – access to ‘foresight’ offered by research community More ‘open’ (transparent), rigorous policy development processes • – early engagement of policy advisers with researchers at ideas stage, pre ‐ policy early engagement of policy advisers with researchers at ideas stage pre policy – ‘Red teaming’ (confidential process within government) – citizens’ juries (public process outside government) – a more level playing field between interests to allow a wider range of ‘voices’ and a more level playing field between interests to allow a wider range of voices and evidence to be considered – clearer articulation of ‘facts’, ‘values’ and ‘interests’ upfront so it is clearer where evidence is and should be used – requirement on government to explain its reasons for decisions especially when requirement on government to explain its reasons for decisions especially when contrary to scientific advice and to represent the evidence accurately

Demand side proposals: ex ante and ex post Demand side proposals: ex ante and ex post • Variants on a ‘NICE’ beyond new drugs and devices – e.g. across all health and social policy, to test and trial ll h lth d i l li t t t d t i l approaches to ‘what works’ (Civil Service Reform Plan, 2012) – e.g. an ‘independent’ body linking supply and demand for evaluation (Puttick, 2012) – e.g. a body separate from Government Departments to commission evaluations of their policies and assess the quality of Departments’ own evaluations (Hallsworth and quality of Departments own evaluations (Hallsworth and Rutter, 2011, p24) – e.g. a body or bodies to undertake independent ex ante & ex post assessments of policy plans (e.g. an extended OBR)

Demand side proposals: ex ante evaluation Range of other controls on policy and accountability for use of evaluation f h l l d b l f f l • – Review of UK ‘Impact Assessments’ to produce closer integration with original Business Cases and reduce post hoc rationalisation g p – Making a reality of plans for ‘Post ‐ Implementation Reviews’ in IAs – Stronger incentives for policy agencies to use evidence routinely • e.g. block on introduction of proposals unless policy makers can show how the evidence relates to or supports their proposals (with rights of appeal if weak evidence) g pp ) – Select Committees to undertake an ‘evidence check’ on all submissions – Publication of all Departmental Risk Assessments of major policies

Demand side proposals: ex post evaluation Greater use of experimentation (RCTs) to inform policy choices ( ) • (Haynes et al, 2012 – Behavioural Insights Team in Cabinet Office) – Bold statements such as Bold statements such as • ‘ RCTs are the best way of determining whether a policy is working ... we should and could use RCTs much more extensively in domestic public policy to test the effectiveness of new and existing interventions to test the effectiveness of new and existing interventions ... (p6) ’ (p6) • Straightforward process summed up by the title ‘Test, Learn, Adapt’ Also wider endorsement of policy ‘pilots’, ‘demonstrations’, • ‘pathfinders’, ‘trailblazers’