London Office of Technology & Innovation Workshop: Evaluation - PowerPoint PPT Presentation

London Office of Technology & Innovation Workshop: Evaluation Framework for Assistive Technology pilots 13 November 2019 Lead boroughs: Newham, Camden and Hackney LOTI_LDN medium.com/loti #LOTI

10:00 Welcome and Introductions Priya Javeri

10:05 1. To learn from boroughs’ experiences of evaluating previous AT trials Objectives for today: 2. To create a template evaluation framework for ATs

9:30 Arrival 10:00 Welcome and Introductions - Priya 10:05 Objectives - Priya 10:10 Exercise 1: Sharing of Experiences - LOTI 10:40 Exercise 2: Creating Personas - LOTI Agenda 11:10 Break 11:20 Recap of LOTI AT evaluation research - LOTI 11:30 Exercise 3: Creating a prototype evaluation framework - Priya 12:25 Wrap up and next steps 12:30 Workshop Close

10:10 Sharing of experiences ● What AT pilots have you run / are you running now? Exercise 1 ● What has been your approach to evaluation? ● What have been the strengths and weaknesses of your evaluations?

10:40 Creating personas ● From your experiences with AT, what factors about a person, their life and context are most important to understand before implementing a new Exercise 2 AT intervention? ● Which factors are common to almost any person and scenario? ● Which factors are unique to certain types of person and scenario?

11:10 Break

11:20 AT evaluation - Research Summary

Key points from interviews with Sutton and Hackney Learnings from existing projects and considerations for future evaluations ● create a baseline ● Set up control groups ● Involve end users ● Gather qualitative and quantitative data ● Consider: a) the technology and any processes in place aspects before commissioning, b) the technical practicalities, c) borough context, d) the purpose Existing tools: LGA’s Digital Social Care Programme and Global Disability Innovation Hub

What we’ve learned from our desk research From other frameworks ● NICE framework ● The Nesta Standards of Evidence ● SETT Framework From case studies: ● RNIB - highlights the need to provide training including takeaway factsheets / user manuals to users of AT devices. ● CQC - highlights that different ATs (independent living, condition management etc.) might require bespoke qualitative and quantitative evaluation.

How LOTI can add value Develop a consistent methodology for evaluating AT pilots that can enable LOTI member boroughs to compare like for like technologies and make better investment decisions.

LOTI’s outcomes-based methodology could be used to help boroughs frame their hypothesis about the intended change they believe an AT might make to help someone live independently. PROBLEM SOLUTION OUTCOME 2 - Discover 3 - Define 4 - Develop 5 - Deliver 1 - Motivate Phase What Generate Focus Generate Focus real-world outcome do we want to What are the real enable? What specific What hypotheses problems and what Which prototypes Key problem(s) will we do we want to test Question do we know about perform best? try to solve? by prototyping? them? (This is not the solution, tech or data!) What problems relate to technology and What role can technology and data play in Tech & Data data? enabling the desired outcome? Non What problems relate to people, processes Beyond technology, what is needed to Tech & and conditions? achieve the desired outcome? Data

LOTI draft AT Evaluation Framework template Desired Real-World Outcome (applying slide 5) : For each of the below, we need to capture: 1) How are things now? What’s our baseline? 2) What’s are the specific problems the AT will address / what are the user needs? 3) What’s our hypothesis for change and how it will we measure it? 4) What results and insights have we captured? INDIVIDUAL ENVIRONMENT Goal: Assess the individual’s environment (including the context of Goal: Assess the individual’s current abilities, unique needs, and any public service support being delivered), how the proposed AT functional areas of concern. integrates, and the individual’s current capabilities within those environments in comparison to other individuals. TASKS TOOLS Goal: Assess what actually happens in the environment (including Goal: Assess the devices, services, resources and strategies that are the context of any public service support being delivered) and the currently in place to support the individual. Assess whether the impact of current demands on the individual. proposed tools are user-centred and/or task orientated. Consider what Standard of Evidence you need to make the desired decision: What are the variables and which can you control? ● ● What are the limitations of your sample size? What qualitative and quantitative data can you realistically capture? ●

11:30 Creating a prototype evaluation framework ● What are the objectives of AT evaluations / what needs must they serve, Exercise 3 individually and collectively? ● Against what headings do we need to capture information? ● What is realistic to measure?

12:25 Next steps ● Agreeing and prototyping the evaluation Wrap up framework ● Refining it for more systematic testing of ATs

12:30 Workshop close

London Office of Technology & Innovation LOTI_LDN medium.com/loti #LOTI