Evaluation: Using CDCs Evaluation Framework By: Thomas J. Chapel, - PowerPoint PPT Presentation

Practical Program Evaluation: Using CDC’s Evaluation Framework By: Thomas J. Chapel, MA, MBA Chief Evaluation Officer CDC/OADPG Tchapel@cdc.gov 404-639-2116

“... If you blindfold someone, put them in a large field, and tell them to walk forward in a straight line. Within minutes and without fail… 2

“... they will walk in a circle. Robert Krulwich, NPR, All Things Considered, 11/23/10 3

Objectives  State CDC Evaluation Framework steps and standards  Describe the added value of strong program description  Describe how to make strong decisions on evaluation focus 4

Defining Evaluation  Evaluation is the systematic investigation of the merit, worth, or significance of any “object” Michael Scriven  Program is any organized public health action/activity implemented to achieve some result 5

Integrating Processes to Achieve Continuous Quality Improvement What do  Continuous Quality we do? Improvement (CQI) cycle  Planning — What actions Why are we will best reach our goals How do we doing and objectives. do it? well or  Performance poorly? measurement — How are we doing? How are we  Evaluation — W hy are we doing? doing well or poorly? 6

 “ Research seeks to prove , evaluation seeks to improve …” M.Q. Patton

Enter the CDC “Evaluation” Framework 8 8

Enter the CDC Good M&E = use of findings Evaluation Framework 9

Enter the CDC Good M&E= use of findings Evaluation Framework Focus is situation -specific 10

Enter the CDC Good M&E = use of findings Evaluation Framework Early steps key to best focus Focus is situation -specific 11

Step 2: Describing the Program: Complete Logic Model Short- Intermediate Long-term term Effects/ Effects/ Inputs Activities Outputs Effects/ Outcomes Outcomes Outcomes Context Assumptions 12

What the program and its staff actually do Short- Intermediate Long-term term Effects/ Effects/ Inputs Activities Outputs Effects/ Outcomes Outcomes Outcomes Context Assumptions 13

Results of activities: Who/what will change? Short- Intermediate Long-term term Effects/ Effects/ Inputs Activities Outputs Effects/ Outcomes Outcomes Outcomes Context Assumptions 14

Finding Activities and Outcomes Plan Actions/ Objectives Goals Tactics LT Eval ST or MT Outcomes Activities Outcomes or Impacts Outcome Measures Process Measures PM Impact Measures Progress Measures Key Performance Indicators Impl. Measures Success Factors 15

Core Funding — Activities and Outcomes Activities Outcomes • • Hire staff and establish org locus People with disabilities are • Develop advisory committee integrated into promotion and • Analyze existing data prevention activities • Evaluate and monitor activities • Plans [of others?] address • Develop strategic planning process needs of people with • Develop strategy disabilities • Develop access strategies • • [Others?] implement access Active collaboration and cooperation of partners and other strategies • [Who?] implements effective health promo • Increased access to promotion and wellness programs and prevention activities • Strong policy [function?] • Prevent secondary conditions • Strong planning function and strategic plan • Improve health and quality of • Strong disability report and dissemination of life of people with disabilities report 16 16

Core Funding — Logic Model Table Activities Outcomes Hire staff and establish Active collaboration People with Prevent secondary org locus and cooperation of disabilities are conditions partners integrated into Develop advisory promotion and Improve health and committee [Who?] implements prevention activities quality of life of people effective health promo with disabilities Analyze existing data and wellness Plans [of others?] programs address needs of Evaluate and monitor people with disabilities activities Strong policy [function?] [Others?] implement Develop strategic access and other planning process Strong planning strategies function and strategic Develop strategy plan Increased access to promotion and Develop access Strong disability report prevention activities strategies and dissemination of report 17 17

Logic Model Table: Module A: Evidence- Based Interventions Activities Outcomes Consult with Facilitate policy Increased social and Improve health and committee [change?] physical accessibility quality of life of PWDs of environments Facilitate physical (home, school, work) access 18 18

Logic Model Table: Module E: Training Activities Outcomes Use/modify existing Offer training Change KAB of curricula (para)professionals in care and health Collaborate with promotion with people with disabilities partners Consult with faculty and staff of PH training programs 19 19

Practical Program Evaluation Setting an Evaluation Focus

Evaluation Focus — Evaluation Can Be About Anything  Evaluation can focus on any/all parts of the logic model  Evaluation questions can pertain to  Boxes---did this component occur as expected  Arrows---what was the relationship between components

Phases and Types of Evaluation Before (More) Program Mature Program New Program Established Stage Program Begins Program Phase FORMATIVE SUMMATIVE Evaluation Needs Process Outcome Impact Type Assessment Evaluation Evaluation Evaluation Is the program Is the program To what extent is (Some) achieving it’s Is the program achieving its the need being met? Questions operating as short-term long-term What can be done to planned? outcomes/ outcomes and Asked address this need? objectives? impacts?? Source: Based on slides from Jennifer Nichols, Porter Novelli

Enter the CDC “Evaluation” Framework 23 23

Setting Focus: Some Rules Based on “utility” standard:  Purpose: Toward what end is the evaluation being conducted?  User: Who wants the info and what are they interested in?  Use: How will they use the info? 24

(Some) Potential Purposes  Show accountability  Test program implementation  “Continuous” program improvement  Increase the knowledge base  Other…  Other… 25

(Some) Potential Purposes  Show accountability  Test program implementation  “Continuous” program improvement  Increase the knowledge base  Other…  Other… 26

Practical Program Evaluation Elaborating Our Program “Theory”— Mediators

Filling in the Blanks…. 28 28

Where “mediators” live in our logic model… Short- Intermediate Long-term term Effects/ Effects/ Inputs Activities Outputs Effects/ Outcomes Outcomes Outcomes Context Assumptions 29

Prevention Program — Simple Logic Model Capacity Building Change Physical Environments Surveillance Communication Prevent and Control Problem Research and Partnership Development Change Social Environments Leadership 30

Prevention Program — Elaborated Logic Model ACTIVITIES OUTCOMES Identify key SURVEILLANCE factors and populations Adopt changes in Propose policy policies, laws and changes Identify modifiable regulations RESEARCH & risk and protective factors and DEVELOPMENT consequences. Change physical Develop/test environment interventions. Create/identify best method and models. Support/develop CAPACITY Change frontline Diffuse supply of Prevent and Adopt practices BUILDING established/ infrastructure. tools, practices and control and programs takes root. Identify skills and programs problem needs. Identify channels, COMMUNICATION Change knowledge, audiences, and key Generate attitudes and beliefs. demand for tools. behavior. Change social environment Identify strategic PARTNERSHIP partners. Access to leaders. Strong Access to partnerships LEADERSHIP Forum for convening. key groups. at all levels. Develop research and other agendas. Increased resources 8/2/2011

“Reality Checking” the Focus Based on “feasibility” standard:  Stage of Development: How long has the program been in existence?  Program Intensity: How intense is the program? How much impact is reasonable to expect?  Resources: How much time, money, expertise are available? 32

Core Funding — Logic Model Table Activities Outcomes Hire staff and establish Active collaboration People with Prevent secondary org locus and cooperation of disabilities are conditions partners integrated into Develop advisory promotion and Improve health and committee [Who?] implements prevention activities quality of life of people effective health promo with disabilities Analyze existing data and wellness Plans [of others?] Mediators Mediators programs address needs of Evaluate and monitor people with disabilities activities Strong policy [function?] [Others?] implement Develop strategic access and other planning process Strong planning strategies function and strategic Develop strategy plan Increased access to promotion and Develop access Strong disability report prevention activities strategies and dissemination of report 33 33