Data Collection Improving Program Effectiveness Through the Use of - PowerPoint PPT Presentation

Data Collection Improving Program Effectiveness Through the Use of Data Presented to: What Works! Summit Sept. 30, 2015

Objectives • Participants will understand the point and purpose of data collection • Participants will understand the use of data to inform business decisions • Participants will be able to identify tools and methods of data collection • Participants will be able to list barriers to accurate data collection

What Is This Data Collection Thing? • Love It? • Measures? • A Pain? • Answers? • Quantitative? • Nonsense? • Outcomes? • Qualitative? • A Ding? • Statistics? • Math? • Don’t care about it? • Information? • A Process? • Evaluation? • Necessary? • Checks & balances? • Variables?

Data Collection Is…… • the process of gathering and measuring information on variables of interest, in an established systematic fashion that enables one to answer stated research question hypotheses, and evaluate outcomes.

This Data Collection Thing is NOT • Love It • Measures A Pain • Answers • Quantitative Nonsense • Outcomes • Qualitative A Ding • Statistics • Math Don’t care Don’t care • Information • A Process about it about it • Evaluation • Necessary • Checks & balances • Variables

Why Collect Data? • If no data is collected, several questions will naturally come to mind when observing how a program operates. – Are the services being provided correctly? – Are enough people being served? – Are the funds for the program being utilized properly? • Without data being collected it is difficult to make determinations on the various aspects of the services being provided.

Consider…. An agency wants to launch a new treatment/service but remains • concerned about staff expansion and utilization – Q. Should we move forward? Community stakeholders want to reduce school drop out and high • crime in a particular area in the city intervention addresses only part of the issue – Q. Are funds being used properly? An agency director knows that treatment results should yield • higher levels of success but something wrong – Q. Why is it not working for the population – Q. What if anything needs to be adjusted or changed?

Why Collect Data! • Data is the life line of any business – It helps one marketing decisions and fine ‐ tune new product launch strategy – Breaks a macro picture into a micro one – Acquires meaningful insights from the dataset – Bases critical decisions on findings – Rules out human bias through proper statistical treatment • Data is a decision supporting system

Improving Program Effectiveness • A program’s effectiveness can be determined through the collection of data. This allows us to see: – What works? How can we continue it? – What isn’t working? How can we improve it?

How to Collect Data Develop time frame to collect data • How often does the data need to be collected? – Repeatable • The mechanism used to collect the data (both the form and the – submission process) must be something that can be used as often as data needs to be collected. Clear and Concise • The requirements of the people generating the data cannot be – so demanding that they struggle to pull the data together or cannot dedicate the time to collect it Hold people turning data in responsible •

Data Collection Methods Include • Face ‐ to ‐ face or telephone interviews • Computer ‐ assisted data sheets • Questionnaires that are either personally, electronically or administered through the mail • Observation of individuals and events with or without videotaping or audio recording

Example Of Computer Assisted Data Sheet • Bob could you insert a DC tracker to show an example of bullet #2

Obstacles to Data Collection • Accuracy • Non – Existent Standards of Protocol • Interpretation • Connected to locally and clearly defined goals • Training • Collection Tools

Consequences from Improperly Collected Data Include • Inability to answer research/impact questions accurately • Inability to repeat and validate interventions • Distorted findings resulting in wasted resources • Misleading others to pursue fruitless avenues of investigation • Compromising decisions for public policy • Causing harm to human participants

Why Protocols Irrelevant or duplicate data collected • Pertinent data omitted • Erroneous or misinterpreted data collected • Too little data acquired from client • Data base format causes disorganized profile • Poor documentation from staff • Conflicting data • Insufficient time • Lack of equipment •

Clearly Defined Goals • Stakeholders goal or expected outcome is too general • Goals or expected outcomes does not derive meaningful results that can inform decisions • Goals or expected outcomes is not measurable or observable • Client, provider, stakeholder not involved – what is measured is of little to no value

Getting to Clearly Defined Goals Make sure you have • A brief description of the project • The specific data that is needed • The rationale for collecting the data • What insight the data might provide (to a process being studied) and how it will help • What will be done with the data once it has been collected

Determining WHAT to Measure Too much data is almost as bad as no data • – Effective measurement is the goal – “Paralysis by analysis” must be avoided Identify data points that touch on all aspects of program operation for • collection – All facets of program operation should be considered (cases being seen, cases completed, staffing, finances, etc.) Make sure that the data requested is data that can be readily obtained • – If a data point requires too much time to collect, the chances of it being consistently collected decrease significantly

Determining HOW to Measure • Once data is obtained, it must be interpreted correctly in order to understand how successful operations are • Reasonable stratifications of performance are important in proper evaluation • Periodic reviews of stratifications must be done to make sure that they are giving the performance evaluations that are desired

Using Data to Make Decisions Having identified the data points to monitor, it is important to use • the data to help make program management decisions Perspective needs to be used to understand trends vs. individual • data points – One must not read too much into small sample sizes Specific events that have taken place in operations (an • employee/therapist leaving the company, etc.) must be considered when interpreting the data

Reporting Data reporting is the process of preparing and disseminating findings to the community Monitor, convert & deploy your data • – Observe for trends – Converted into more understandable and precise presentation formats, such as charts, files or graphs – Create written data report – Distribute report to those parties that need it Determining your audience • – Whenever reporting out on data, it is crucial that you understand who is going to be reviewing the reporting – Understand that different levels of staff maybe interested in different levels of detail

When Reporting • Beware of External Influences that could skew report – Professional competition – Job security – Unclear guidelines from funding sources – Publication pressure • Beware of Internal Influences that could skew report – Individual ego or vanity – Personal financial gain – Incompetence – Sloppy writing/reporting

Be Sure to Report • If your intervention/ program had exactly the effects you expected and hoped it would. • If your program/ intervention had no effect. • If your program/ intervention had a negative effect. • If your program/ intervention had the effects you hoped for and other effects as well. • If your program / intervention gives you a clear indication that what you’re doing is accomplishing your purpose

QUESTIONS? Leslie Byam, DC Project Director, EBA, lbyam@ebanetwork.com • Bob Sayles, IT Director, EBA, bsayles@ebanetwork.com • Leslie Serena, Executive Director, Camelot Community Care, • LSerena@camelotcommunitycare.org Jennifer Bonness, Camelot Community Care, Clinical Director, • JBonness@camelotcommunitycare.org