Evaluation at PCORI Michele Orza, ScD, Senior Advisor to the Executive Director PCORI Board of Governors Meeting Alexandria, VA May 2014 Board of Governors Meeting, May 5, 2014 80
Objectives The objectives of this presentation are to provide: An introduction to Evaluation at PCORI A progress report on some core activities A guide to where to find further information and opportunities to provide input Board of Governors Meeting, May 5, 2014 81
Our Current Tasks Building our Evaluation Framework , that is, delineating, organizing, and prioritizing the questions that staff and stakeholders have about PCORI’s work and about PCOR in general Determining how we can measure some of the elements that we know already will be central to answering many of the questions, especially our three goals and Engagement in Research Undertaking many evaluation activities that have clearly emerged as high priorities, such as surveys and assessing the impact of Engagement on the early phases of research Board of Governors Meeting, May 5, 2014 82
What Is Our Evaluation Framework? Our framework organizes our evaluation questions and describes how we will answer them Evaluation Metrics/ Methods Sources Questions Indicators What do For each What From where PCORI staff evaluation approach will will we get the and question, what we take to data to answer stakeholders are we answering each want/need to measuring and each evaluation know about how will we evaluation question? PCOR and measure it? question? PCORI? Board of Governors Meeting, May 5, 2014 83
Our Evaluation Framework Includes Three Kinds of Questions 2 1 Are we accomplishing our What are we doing? goals? Are we doing it efficiently and Producing useful information? effectively? Speeding its uptake? Influencing research? Are we on track? 3 How do the various components of PCORI’s approach contribute to reaching its goals and achieving its mission? Board of Governors Meeting, May 5, 2014 84
1: Questions about Our Day-to-Day Work 2 1 Are we accomplishing our What are we doing? goals? Are we doing it efficiently and Producing useful information? effectively? Speeding its uptake? Influencing research? Are we on track? 3 How do the various components of PCORI’s approach contribute to reaching its goals and achieving its mission? Board of Governors Meeting, May 5, 2014 85
Legend Our Vision: Patients and the public have information they can use to make decisions Coming Soon Target that reflect their desired health outcomes. Board of Governors On Track Off Track FY2014 Dashboard Previous Actual Our Goals: Increase Information, Speed Implementation, and Influence Research (As of 3/31/2014) Needs Attention Current Actual Broad Awards Summary Funded Research Portfolio Methodology Standards Dissemination & Implementation Px Dx Tx Screening Other Apps % of LOIs Awards % of Apps Metric Q4 Q1 Q2 Q3 60 100 Resubs % of Apps Resubs % of Awards Application Adherence (%) NA NA 74% Percent Downloads from Web (#) 80 40 Citations (#) Coming Q3 60 Endorsements (#) 20 Adoption by Others (#) 40 Pipeline to Proposal Awards 0 20 Cycle I Cycle II Cycle August Winter Spring 65 III 2013 2013 2014 Q1, Q2 Q3, Q4 0 Priority Topics Pipeline – Funded by Q4 2014 Cycle I Cycle II Cycle III Aug 2013 All Engagement Event Survey Results Under Consideration Currently Posted Funded Progress of Projects 86% of participants say that they have done Pragmatic Studies = 15 0/0 Q1 Q2 Q3 Q4 100 something new with PCOR since the workshop 40+ % Meeting Milestones Targeted PFA = 3 3/6 80 Ambassadors Fully Trained 67 63 Usefulness: Coming by Q4 100 60 Engagement Impact: Coming by Q4 Q3 Q1/Q2 Other Projects 40 Coming Q3 Completion of Phase I of PCORnet 2014 Research Funding Commitment 20 1/3 0 $528M Research Pipeline PPRNs CDRNs Q3 Q1 Q2 Q2 Type of Project Communications & Ops (% of Target or Reference) Funds Committed for External Research 2014 Staffing Plan – Number of People 350 Q1 Q2 Communications Q4 Targeted 165 300 Unique Web Visitors (Same Q year-ago) 134 151 154 250 Email Click-thru Rate (Industry Standard) 142 150 143 Broad Q1 Q2 Q3 200 Journal Articles (number, no target) 8 12 10 150 $M PCORnet 2014 Expenditures – $M Media Mentions (Same Q year-ago) 450 333 422 100 Operations Pilots $183M 50 Award to Contract Time (w/in 90 cal. days) 80 16 37 0 Q1 Q2 Q3 Contracts Response Time (w/in 2 bus. days) 85 96 99 FY 2012 FY 2013 FY 2014 (as of 2/28/14) Science Response Time (w/in 3 bus. days) 54 63 81
2: Questions about Our Goals 2 1 Are we accomplishing our What are we doing? goals? Are we doing it efficiently and Producing useful information? effectively? Speeding its uptake? Influencing research? Are we on track? 3 How do the various components of PCORI’s approach contribute to reaching its goals and achieving its mission? Board of Governors Meeting, May 5, 2014 87
First Goal: Useful Information Substantially increase the quantity, quality, and timeliness of useful, trustworthy information available to support health decisions Board of Governors Meeting, May 5, 2014 88
Draft Usefulness Criteria We are developing criteria to assess the potential usefulness of the information we produce: We are refining Rationale/User-Driven these criteria, People who would use the information have been identified cross-walking Specific uses for the information have been identified them with our People who would use the information are asking the question other criteria (such as for Merit Research Question/User-Focused Review), and pilot Study compares options that are relevant for the people who testing them, and would use the information will soon apply Study assesses the outcome(s) that matter for the people who them to our would use the information portfolio of funded Real-world Use studies. Results could provide a clear answer to the question Results could be tailored to individuals or subgroups Results could be scaled/spread beyond the study setting Board of Governors Meeting, May 5, 2014 89
Usefulness: What could we say about it in 2014? How our portfolio stacks up against our criteria for potential usefulness and How applications we funded compare to those we didn’t How studies funded in earlier cycles compare to later ones How studies funded under Broad PFAs compare to Targeted How our portfolio compares to others on these criteria What patients and stakeholders think about the potential usefulness of our portfolio For example, what concerned patients and relevant stakeholders think about the cluster of asthma, pediatric, or mental health studies we are funding Board of Governors Meeting, May 5, 2014 90
Assessing Potential and Actual Usefulness How do the Is the Do people find studies we fund information from information from measure on PCORI studies PCORI studies usefulness being used? By useful? criteria? whom? How? Refine criteria and incorporate into funding decisions Board of Governors Meeting, May 5, 2014 91
Second Goal: Use of PCORI Information Speed the implementation and use of patient- centered outcomes research evidence Board of Governors Meeting, May 5, 2014 92
Draft Measures of Use Impact: Changes Dissemination Uptake and Use in Health Decisions or Care (Measure for all PCORI (Measure for a subset of funded studies) PCORI funded studies) (Measure for small set of exemplar studies) • Results reported back to study • Adoption of study findings in the • Change in health decisions participants study setting or health care among key • Access to PCORI study report • Incorporation into: groups (patients, clinicians, • Systematic reviews • Presentations: payers, etc.) • Patient and consumer education • Scientific/professional audiences materials • Lay audiences • Graduate Medical Education • Bibliometrics: (GME) or Continuing Medical • # of Publications Education (CME) • Time to publication Note: Most of these metrics are typically • Practice guidelines • Impact factor not measurable until after study • Decision making infrastructure • Citations completion, and in many cases, are (e.g. electronic decision aids, • Alternative metrics for key groups typically not measureable until several clinical reference tools) (patients, clinicians, payers, etc.): years after study completion. • Payer policies • # manuscript downloads • Institutional, local, state, and • # manuscript bookmarks national policy • Media coverage • Social media coverage Board of Governors Meeting, May 5, 2014 93
Third Goal: Influence Research Influence clinical and health care research funded by others to be more patient-centered Board of Governors Meeting, May 5, 2014 94
Draft Measures of Influence We are now or soon will be measuring: Endorsement, Promotion, and Dissemination of PCORI work Use of PCORI Methodology Standards for Patient-Centeredness Use of PCORI approaches: Topic Generation and Research Prioritization Merit Review Engagement Communication and Dissemination Use of PCORI guidance re: Patient-Centered CER Use of PCORI-supported curricula or training Collaborations/Co-funding with other funders We will have to wait a few more years to measure: Use of PCOR Methods evidence Use and support of PCORnet Board of Governors Meeting, May 5, 2014 95
Recommend
More recommend
