Creating a One-Page Evaluation Case Summary: Context, Narrative, - PowerPoint PPT Presentation

Creating a One-Page Evaluation Case Summary: Context, Narrative, and Style Strategies Lynne Page Snyder, Ph.D., MPH Principal Research Scientist NORC at the University of Chicago AcademyHealth Annual Research Meeting June 25, 2018, Seattle, WA

Overview Challenge = reporting that speaks to its audience  Audience: who are they and what are their concerns?  Context: framing the story  Narrative: for CMMI evaluations, writing about numbers  Style: guidance for drafting  Example: one-page visual case summaries for first-round Health Care Innovation Award (HCIA) evaluation  For more information… Acknowledgments: Health Care team at NORC; Mitali Dayal (CMMI). 2

As LBJ Once Said, “A Range is for Cattle,…” Source: Kat Jayne from Pexels. 3

“Give Me A Number.” Source: Ryan McGuire, https://gratisography.com/. 4

One Page Case Summary for HCIA Evaluation  Evaluation of first-round Health Care Innovation Award (HCIA), High-Risk/Patient Targeting Portfolio  Range of models  Shared focus on highest-risk beneficiaries (Medicare, Medicaid)  Mixed methods  Client = CMMI  The ask = one page visual case summary for each of 23 cases (awardees), akin to a briefing document  Constraints : time, budget, production resources (software), statement of work (deliverable specifications)  Flexible design to enable consistency across diverse cases 5

More About Goals for the One- Pager…  Intended to convey part of a report, rather than be summative or comprehensive  Free-standing rather than a preview. Audience likely will not read the report and may use the one pager to make a decision  Informed by substantial literature on how to transform evaluation reporting from chore (and bore) to engaging and effective communication 6

What Our Evaluation One- Pager is not…  Data visualization or infographic (greater reliance on text)  Dashboard (one-time, not intended for update)  Peer-reviewed article (less standardized)  Social media post (audience = client) 7

Audiences Policy influentials & decision makers Media & Contracting social media Representative, Client Patients,  What does our audience most families, & want to know? Health caregivers plans & payers Health care: clinicians, practices, hospitals, skilled Congress, the State & local nursing facilities courts, executive government, branch agencies civic groups 8

Context That the Audience Sees  Define the situation  What do we know? What don’t we know? What do we assume?  Analogies: What is similar? What is different?  Identify decision makers’ concerns  What’s the objective?  What’s the story? Timeline? The facts?  What are the options for action? What is feasible for now?  What expectations about causes and effects make certain options preferable to others? What new evidence might change or challenge a presumption?  Add context: people and organizations as actors moving through time Source: Neustadt & May, Thinking in Time 9

More About Narrative: Telling a Story Using Numbers “Good statistics involves principled argument that conveys an interesting and credible point.”  Making comparisons and explaining basis for same  Testing claim(s)  Respecting chance (null hypothesis)  What makes an argument persuasive?  Effect size  Narrative detail and style  Generalizability  Being sufficiently important and compelling to change minds  Credible Sources: Abelson, Statistics as Principled Argument 10

Writing About Numbers For multivariate analysis:  Set context  Choose examples and analogies  Tailor vocabulary to audience  Decide whether and how to use text, tables, and/or figures  Interpret number(s) in the text  Specify both direction and size of association between variables  Describe overall patterns, illustrate with examples, and note exceptions Sources: Miller, Chicago Guide to Writing About Multivariate Analysis 11

Style: Best Practices for Reporting and Visualizations  Graphics  Keep visuals simple (images, graphs)  Reduce clutter  Refer to visualization checklist(s)  Type  Communicate through font style and sizes  Develop a style sheet  Color  Keep the scheme simple  Plan for 508 compliance  Arrangement  Follow design principles for layout, grouping, use of white space, alignment  Present no more than 5-7 points per section/page  Use consistency (and breaks from consistency) to convey priorities Source: Stephanie D.H. Evergreen & Ann Emery, Reporting & Evaluation Report Layout Checklist, in Evergreen, Presenting Data Effectively 12

Big Decisions about One Pager  Frame/Context = COR, CMS front office  Narrative =  modest and limited positive findings on promising models  findings based on data from interviews/site visits, program documents, and claims  analyses include multivariate regression models, difference-in- differences, sample surveys, themes in qualitative data  Style = free-standing, selectively comprehensive  Importance of client review (akin to pretest) 13

What’s on the One -Pager? Basic information  About awardee: awardee name, summary description, classification of model type  About cooperative agreement: award $$, performance period, reach, populations served, data sources for evaluation Evaluation Domains 14

Basic information Key findings Secondary findings

Footnote

Key findings

For more information,  Robert P. Abelson. Statistics as Principled Argument . Hillsdale, NJ: Lawrence Erlbaum Associates, 1995.  Robert E. Stake. The Art of Case Study Research. SAGE Publications, 1995.  Ann Emery. Website on Information Design, at http://annkemery.com/ .  Stephanie Evergreen. Presenting Data Effectively . SAGE Publications, 2014; and website, Evergreen Data: Intentional Reporting and Data Visualization , Source: Kat Jayne from Pexels. at http://stephanieevergreen.com/ . Jane E. Miller. Chicago Guide to Writing about Multivariate Analysis , 2 nd edition. Chicago: Univ.  Chicago Press, 2013.  Richard E. Neustadt and Ernest R. May. Thinking in Time. The Uses of History for Decision Makers . Free Press, 1988.  NORC at the University of Chicago. HCIA Complex/High-Risk Patient Targeting: Third Annual Report . 2017. At https://downloads.cms.gov/files/cmmi/hcia-chspt-thirdannualrpt.pdf . 21

Lynne Page Snyder, Ph.D., MPH snyder-lynne@norc.org Office: 301.634.9569 Cell: 301.520.1809 Thank You!