OUTCOME evaluation step by step The webinar will begin at 1 p.m. - PDF document

WEBINAR Outcome Evaluation: Step ‐ by ‐ Step 3/21/2019 OUTCOME evaluation step ‐ by ‐ step The webinar will begin at 1 p.m. Eastern time Introductions Christina Titus Mike Lesiecki Lori Wingate evalu ‐ ate.org 1

WEBINAR Outcome Evaluation: Step ‐ by ‐ Step 3/21/2019 www.connectedtech.org/ccta.html | atecenters.org/ccta Collin College (lead) Florence ‐ Hillsborough Darlington Community College Technical College City College of San Francisco evalu ‐ ate.org 2

WEBINAR Outcome Evaluation: Step ‐ by ‐ Step 3/21/2019 www.atecentral.net Webinars Resource Blog ATE Survey Data Library www.evalu ‐ ate.org evalu ‐ ate.org 3

WEBINAR Outcome Evaluation: Step ‐ by ‐ Step 3/21/2019 Materials Slides Handout Recording www.evalu ‐ ate.org/webinars/mar19 This material is based upon work supported by the National Science Foundation under grant number 1600992. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the presenters and do not necessarily reflect the views of NSF. evalu ‐ ate.org 4

WEBINAR Outcome Evaluation: Step ‐ by ‐ Step 3/21/2019 Process v. Outcome Evaluation Evaluation of the activities that Determination and evaluation a project carries out of the changes a project brings and the materials or products it about creates or uses in service delivery Process v. Outcome Evaluation ● Quality of program CHANGES in content Attitudes ● Quality of program Knowledge materials or facilities Skill ● Extent of reach to Competence intended and other audiences Behavior ● Adequacy and logic of Social or economic program design conditions ● Level of participant satisfaction evalu ‐ ate.org 5

WEBINAR Outcome Evaluation: Step ‐ by ‐ Step 3/21/2019 Outcome Evaluation Steps 1. Define intended outcomes 2. Identify evaluation questions 3. Plan for data collection and beyond 4. Collect and analyze data 5. Interpret results (answer evaluation questions) Webinar Sections Define intended outcomes Identify evaluation questions Plan for data collection and beyond Collect and analyze data Interpret results (answer evaluation questions) evalu ‐ ate.org 6

WEBINAR Outcome Evaluation: Step ‐ by ‐ Step 3/21/2019 Define Intended Outcomes and Identify Evaluation Questions Outcome Any chang e resulting from project activities and outputs evalu ‐ ate.org 7

WEBINAR Outcome Evaluation: Step ‐ by ‐ Step 3/21/2019 Activity What a project does, the actions it takes Goal An achievement being sought May focus on activities or outcomes evalu ‐ ate.org 8

WEBINAR Outcome Evaluation: Step ‐ by ‐ Step 3/21/2019 Activity goal Outcome goal (what a project will do) (what difference it will make) The project will deliver Webinar participants will four webinars per year, improve their evaluation serving 1,000 people. knowledge and practices. Real goal statements from real NSF ‐ funded projects evalu ‐ ate.org 9

WEBINAR Outcome Evaluation: Step ‐ by ‐ Step 3/21/2019 The goal of the project is to increase the supply of qualified cybersecurity professionals for industry and government. Outcome: More qualified workforce The goal of this project is to develop an associate's degree in mechatronics, incorporating pathways from local high schools into the degree offering at three partner colleges. Activity: Create degree program evalu ‐ ate.org 10

WEBINAR Outcome Evaluation: Step ‐ by ‐ Step 3/21/2019 This project has the overarching goal of increasing awareness of opportunities in science, technology, engineering, and mathematics (STEM) disciplines for women and underrepresented minorities. Outcome: Change what people know about STEM disciplines The project's goal is to build a sustainable program to enhance process technology education by introducing new hands ‐ on opportunities through use of light ‐ weight extremely low ‐ cost miniature industrial equipment with a small footprint that fits on a standard desktop or which can be taken home for use in homework assignments. Activity: Create program, use new equipment evalu ‐ ate.org 11

WEBINAR Outcome Evaluation: Step ‐ by ‐ Step 3/21/2019 INTENDED OUTCOMES specific , realistic statements about what is expected to change for individuals or groups relevant to the need that the project is designed to address Current wind energy workforce: CASE Growing a New Generation of Energy Technicians and Professionals evalu ‐ ate.org 12

WEBINAR Outcome Evaluation: Step ‐ by ‐ Step 3/21/2019 Projected retirement within 10 years: CASE Growing a New Generation of Energy Technicians and Professionals 1) Increase 2) Design and 3) Enhance academic rigor activate career recruitment, pathways retention, and placement efforts CASE Growing a New Generation of Energy Technicians and Professionals evalu ‐ ate.org 13

WEBINAR Outcome Evaluation: Step ‐ by ‐ Step 3/21/2019 Project Goals 1. Improve and expand academic rigor and relevance across core technology curriculum and wind energy technology ‐ specific curriculum. 2. Design and put into action wind/renewable energy career pathways. 3. Enhance and expand recruitment, retention, and placement efforts across technology programs. Project actions = Activities Logic models are a great tool for evaluation planning! evalu ‐ ate.org 14

WEBINAR Outcome Evaluation: Step ‐ by ‐ Step 3/21/2019 evalu ‐ ate.org 15

WEBINAR Outcome Evaluation: Step ‐ by ‐ Step 3/21/2019 evalu ‐ ate.org 16

WEBINAR Outcome Evaluation: Step ‐ by ‐ Step 3/21/2019 Focus of OUTCOME EVALUATION evalu ‐ ate.org 17

WEBINAR Outcome Evaluation: Step ‐ by ‐ Step 3/21/2019 Question 1 Outcome Evaluation evalu ‐ ate.org 18

WEBINAR Outcome Evaluation: Step ‐ by ‐ Step 3/21/2019 Question 2 Outcome Evaluation Question 3 Outcome Evaluation evalu ‐ ate.org 19

WEBINAR Outcome Evaluation: Step ‐ by ‐ Step 3/21/2019 Question 4 Outcome Evaluation Summary  Clearly define intended outcomes.  Identify multiple levels of outcomes.  Frame evaluation questions around outcomes.  Ask evaluation questions that allow for a range of conclusions.  Bonus: Always include an evaluation question like this: “What are the project’s unintended positive or negative side effects or outcomes, if any?” evalu ‐ ate.org 20

WEBINAR Outcome Evaluation: Step ‐ by ‐ Step 3/21/2019 Resources Getting to Outcomes TM Logic model template, online course, and more Evaluation Questions Checklist Book chapter by Michael Quinn Patton on defining outcomes Planning for Data Collection and Beyond evalu ‐ ate.org 21

WEBINAR Outcome Evaluation: Step ‐ by ‐ Step 3/21/2019 For each evaluation question, specify: Indicators What will be measured in order to answer evaluation questions evalu ‐ ate.org 22

WEBINAR Outcome Evaluation: Step ‐ by ‐ Step 3/21/2019 Data Sources & Methods Where information related to indicators will be obtained and how People Who will be responsible for which aspects of data collection evalu ‐ ate.org 23

WEBINAR Outcome Evaluation: Step ‐ by ‐ Step 3/21/2019 Timing When data will be collected and with what frequency Analysis How collected data will be transformed into usable information evalu ‐ ate.org 24

WEBINAR Outcome Evaluation: Step ‐ by ‐ Step 3/21/2019 Interpretation How evaluation findings will be translated into conclusions For each evaluation question, specify:  Indicators  Data sources and methods  People  Timing  Analysis  Interpretation evalu ‐ ate.org 25

WEBINAR Outcome Evaluation: Step ‐ by ‐ Step 3/21/2019 A matrix is a great way to show relationships between data collection plan elements Outcome Evaluation Question 1: To what extent are students using career pathways established by the project? evalu ‐ ate.org 26

WEBINAR Outcome Evaluation: Step ‐ by ‐ Step 3/21/2019 Outcome Evaluation Question 1: To what extent are students using career pathways established by the project? what will be how data how results will be used to answer evaluation measured will be obtained questions The evaluation will include a survey of students and secondary analysis of institutional data. evalu ‐ ate.org 27

WEBINAR Outcome Evaluation: Step ‐ by ‐ Step 3/21/2019 The evaluation will include a But what will survey of students and be measured? secondary analysis of institutional data. The evaluation will include a INDICATOR DATA SOURCE & METHOD survey of students and Number of high Institutional data school students in dual enrollment courses secondary analysis of Number and Survey of dual ‐ percentage of dual ‐ enrolled students institutional data. enrolled students who intend to pursue degree and certificate programs evalu ‐ ate.org 28

WEBINAR Outcome Evaluation: Step ‐ by ‐ Step 3/21/2019 If an “outcome” is not caused by the intervention, it is NOT an outcome. It’s merely a coincidence. —Jane Davidson Outcome change resulting from project activities effect cause or contributor evalu ‐ ate.org 29