Are we making a difference? Lessons from evaluating civil - PowerPoint PPT Presentation

Are we making a difference? Lessons from evaluating civil society-led action on food

• Dr Hannah Pitt • Sustainable Places Research Institute, Cardiff University • Social scientist • Food sustainability, gardening & growing • Research in partnership with civil society organisations

What is evaluation? Monitoring = What are we doing? (outputs) Evaluation = How have we done? (outcomes + process)

Why evaluate?

Aims : • to present evidence of the programme impacts; • to reveal and document how impacts are achieved; • to support and facilitate sharing of learning ; and • to advise on research and evidence elements of programme development.

Process Tender Project Awarded Agree Collect & Report funded Contract plan data selection

Approach Theory of change / logic model = how you expect outcomes to occur over the short, medium and longer term http://bhfood.org.uk/wp-content/uploads/2018/11/Outcomes- and-impact-framework.pdf

e.g. Food Growing Schools London ”the ambition to inspire and equip every school in London to grow their own food”

Outcomes & Indicators Outcome Indicators Collection Method Through involvement in I 1. Head teachers report Annually School survey food growing activities at an increase in positive school, children and behaviour, attention or young people will have attainment: 40% of gained life skills, respondents knowledge and confidence in food growing. I 2. Number of pupils Annually School survey physically involved in food growing: 10% increase over baseline I 3. Number of pupils Annually Pupil survey reporting increased knowledge, skills and confidence in food growing: 20% increase over baseline

Data collection 1. What change are we trying to demonstrate and understand? 2. What will indicate or measure change? numbers & stories 3. Who can help us understand change? e.g. questionnaires/surveys, interviews, focus groups

Case Studies Food Growing Schools London report: https://www.foodgrowingschools.org/resources/files/F GSL_InterimReport_2016_v2.pdf Academic publication on Food Growing Schools London: Pitt, H. , Jones, M. and Weitkamp, E. (2018) Every city a food growing city? What food growing schools London reveals about city strategies for food system sustainability. Sustainability , 10 (8). p. 2924. ISSN 2071- 1050 Available from: http://eprints.uwe.ac.uk/38053

Food For Life https://www.foodforlife.org.uk/

Food For Life Do pupils consume more fruit and vegetables in schools engaged with FFL than pupils in schools not engaged with FFL? Day in the Life Questionnaire : Health Educ Res. 2002 Apr;17(2):211-20 Development and validation of the Day in the Life Questionnaire (DILQ) as a measure of fruit and vegetable questionnaire for 7-9 year olds. Edmunds LD 1 , Ziebland S.

All Food For Life evaluation reports available at: https://www.foodforlife.org.uk/abou t-us/our-impact/evaluation-reports Academic papers:

What is the value of investing in Food For Life?

Social Return on Investment • Aims to capture the full the value of investment beyond market prices • Quantifies the relative importance people place on change they experience • Measured from the perspective of those affected SROI = value of benefits Value of investments

Outcome Indicator Financial Proxy Data source Curriculum Value of staff time Cost per head of Cost half day in- development linked spent in curriculum half day local school training to local issues development work authority area- session £30 per based training head. session, based http://www.aqa.or upon attendance of g.uk/professional- 10 trainees. Plus development/in- teacher cover school-training. supply costs

What is the value of investing in Food For Life? Report on SROI Results: https://www.foodforlife.org.uk/~ /media/files/evaluation%20repo rts/4foodforlifelcssroifullreportv 04.pdf

Academic publications on Food For Life: • Gray, S., Orme, J., Pitt, H., Jones, M. (2017) Food for Life: evaluation of the impact of the Hospital Food Programme in England using a case study approach Journal of the Royal Society of Medicine Open 8 (10)1-9 DOI: 10.1177/2054270417712703 • Jones, M., Pitt, H., Oxford, L., Bray, I., Kimberlee, R., Orme, J. (2017) Association between Food for Life, a Whole Setting Healthy and Sustainable Food Programme, and Primary School Children’s Consumption of Fruit and Vegetables: A Cross -Sectional Study in England. International Journal of Environmental Research and Public Health ,14, 639. ISSN 1660-4601 • Gray, S., Jones, M., Means, R., Orme, J., Pitt, H., and Salmon, D. (2017) Inter-sectoral Transfer of the Food for Life Settings Framework in England. Health Promotion International , pp.1-10. ISSN 0957-4824 • Gray, S., Means, R., Orme, J., Pitt, H., Jones, M. and Salmon, D. (2015) Improving hospital food: Evaluating the impact of the UK Food for Life Partnership. European Journal of Public Health , 25 (Suppl3). p. 380. ISSN 1101-1262 • Weitkamp, E., Jones, M., Salmon, D., Kimberlee, R. and Orme, J. (2013) Creating a learning environment to promote food sustainability issues in primary schools? Staff perceptions of implementing the food for life partnership programme. Sustainability , 5 (3). pp. 1128-1140. ISSN 2071-1050 • Orme, J., Jones, M., Salmon, D., Weitkamp, E. and Kimberlee, R. (2013) A process evaluation of student participation in a whole school food programme. Health Education , 113 (3). ISSN 0965-4283 • Kimberlee, R., Jones, M., Orme, J. and Salmon, D. (2013) Whole school food programmes and the kitchen environment. British Food Journal , 115 (5). pp. 756-768. ISSN 0007-070X • Jones, M., Dailami, N., Weitkamp, E., Kimberlee, R., Salmon, D. and Orme, J. (2012) Engaging secondary school students in food-related citizenship: Achievements and challenges of a multi-component programme. Education Sciences , 2 (2). pp. 77-90. ISSN 2227-7102 • Jones, M., Dailami, N., Weitkamp, E., Kimberlee, R., Salmon, D. and Orme, J. (2012) Food sustainability education as a route to healthier eating: Evaluation of a multi-component school programme in English primary schools. Health Education Research , 27 (3). pp. 448-458. ISSN 0268-1153 • Jones, M., Weitkamp, E., Kimberlee, R., Salmon, D. and Orme, J. (2012) Realizing a holistic approach to food through school gardens and growing activities. Children, Youth and Environments , 22 (1). pp. 75-98. ISSN 1546-2250

Growing Together Does involvement in gardening improve mental health? Evaluation Report available: Jones, M. and Pitt, H. (2016) Growing Together: A mixed methods evaluation of a community gardening project delivered by a social housing association. Year 1. Project Report. UWE Bristol, Bristol. Available from: http://eprints.uwe.ac.uk/28428

Warwick Edinburgh Mental Wellbeing Scale Guidance on the scale: https://warwick.ac.uk/fac/sci/med/research/platform/wemwbs /

Food Power “Food Power aims to strengthen the ability of local communities to reduce food poverty, through solutions developed in partnership and with the support of peers from across the UK. Our goal is to transform the way that people experiencing food poverty access support so they can create long-term, sustainable lives that are free from hunger .” https://www.sustainweb.org/foodpower/

Food Power Are local food poverty alliances making a difference? Year 1 Evaluation Report available: https://www.sustainweb.org/foodpower/publi cations/

Collective impact • Collective impact is an approach to understanding change related to complex problems. • Considers organisations working together rather than in isolation. • A structured way of understanding and measuring collaboration and its impacts. • Research identified 5 conditions of success.

Evaluating Collective Impact dfdvdv How: • Promotes continuous learning • Seeks to understand progress (performance, outcomes) and how & why progress (process) • Uses understanding of collective impact processes to devise a framework for measurement and evaluation. • Evolves over time • Early years - focus on what needs to happen & establishing 5 core conditions • Middle years – increasing focus on impacts, how well is it working • Later years – focus on goals and outcomes, what difference did it make Guide available: https://www.collectiveimpactforum.org/resources/guide- evaluating-collective-impact 30