National Aeronautics and Space Administration Assessing the Impact of NASA’s STEM Engagement Investments: Development of External & Internal Performance Measures March 20, 2018 NASA Glenn Research Center Rob LaSalvia, Center Education Director Rick Gilmore, Evaluation Manager Dr. Catherine Graves, Lead Evaluation Specialist Dr. Tara Strang, Evaluation Specialist
Performance Assessment Strategy, Framework, & Measures Development Process
Assessing the Impact of NASA’s STEM Engagement Investments: Performance Assessment & Evaluation Overview •Broad strategic goals designed to advance NASA’s mission and Strategy address relevant national problems, needs, challenges and Performance opportunities. Performance Assessment Framework •Timeframe: 2018 - 2022 •Ongoing monitoring and reporting of program accomplishments, particularly progress toward pre- established goals. Performance •Includes outputs such as: demographics, course Assessment development, paper/presentation; and outcomes such as building research capacity of higher education institutions. Evaluation •Timeframe: Short term (Annual) •Systematic study using research methods to collect and analyze data to assess how well a program is working and why. Evaluation •Includes outcomes such as: science identity, scientific literacy, STEM graduate program matriculation/career outcomes. •Timeframe: Long term
External and Internal Performance Measures Development Process STEP ONE STEP TWO STEP THREE Benchmark Other Federal Finalized Candidate Review Past Performance Agencies, Literature Review, Performance Measures & Stakeholder Discussions A brief history of NASA’s Office of Summary of guidance from Recommended External and Internal Education performance measures, Performance Measures, literature and practice data collection processes and tools, and (BSA priorities, literature review, Data Collection Sources, Processes, and Tools performance results benchmarking, internal and external to Capture and Report Performance Data stakeholder discussions) • Recommended external and internal FY 2014 – FY 2017 Performance Goals, BSA priorities performance measures • • Annual Performance Indicators, Results • Literature review findings • Data collection sources • Strategic Object Annual Reviews (SOAR) • Benchmarking findings • Data collection processes • OMB and BSA guidance and • Internal stakeholder discussions • Data collection tools recommendations: Concerns, need for • Mission Directorate discussions • Assessing and analyzing performance data Expert Review Panel - External (including ERP procedure and scorecards) change/reformed measures, processes, • and tools stakeholder discussions • Reporting performance data
External and Internal Performance Measures Development Schedule Performance Measures Development Milestones Oct. Nov. Dec. Jan. Feb. Mar. Apr. May Jun. Jul. Aug. Sep. 1. Internal Assess of the Office of Education’s Evaluation & 1 Performance Measurement including pervious APR’s, SOAR assessments, & OMB guidance (October 2017). 2. Review of relevant literature (evidence-based practices & current 2 trends in STEM education and evaluation) & Benchmarking of Federal Agencies’ STEM education performance measurement strategies and tools (November 2017 – March 2018). Initial Input 3 3. Internal NASA Stakeholder Discussions (March 2018). Requested Alignment with OSE Strategy Development 4 4. Expert Panel Review (April 2018). Follow-Up Input 5 5. Draft External and Internal Performance Measures Document, Requested including data collection tools and processes (May 2018). 6 6. Final External and Internal Performance Measures Document, including data collection tools and processes (June 2018). 7 7. (anticipated) Candidate FY19 & FY20 Performance Goals and Annual Performance Indicators due to OCFO-SID (July 2018).
Gathering Input from Internal Stakeholders and External Experts
Internal Stakeholder Discussions Feb. 27 – Mar. 5 Mar. 12 – 23 • Kris Brown • Center Education Directors & Deputy Directors • Joeletta Patrick (MUREP) Mission Directorates Education Leads • • Chris Fitzsimonds (OCFO-SID) • MIRO PIs • Carolyn Knowles (NIFS) EPSCoR PIs • • Jeppie Compton/Crystal Bassett (EPSCoR) • Selected Activity Managers and Third-Party Evaluators • Joeletta Patrick/Erica Alston (Space Grant) with demonstrated success in evaluation and • Diane Detroye (SEAP/NGS) performance assessment • Roosevelt Johnson (Education Senior • SME’s (Mission - Technical People with a History of Advisor) working with Education) • Bev Girten (Informal) • Holly Degn (Budget) • Tammy Rowan (IT Infrastructure) • HQ Strategy Meeting Attendees • National Space Grant Directors’ Meeting - Flip Session Discussions
External Expert Review Panel Discussions 1. Expert Review Panel to be held in Washington, D.C. late April or early May 2. Process for Identifying Candidates for Expert Review Panel (ERP) • Representatives from the NAC and NSF along with evaluation experts were provided • Nationally Recognized Experts with publication records and experience in our needed areas of expertise Program Structure and Evaluation Higher Education Diversity Building Technical Research Capacity IT Systems / Social Media & Emerging Technologies Science Literacy / Large-scale Public Engagement Campaigns • Experience in developing national policy, strategies, work with federal-funded programs
Office of STEM Engagement Performance Assessment Framework DRAFT
Proposed Office of STEM Engagement Performance Assessment Strategy Framework Traditional Performance Assessment Strategy Enhanced Performance Assessment Strategy 2018 NASA Strategic Plan 2018 NASA Strategic Plan Strategic Goal 3: Address National Challenges & Catalyze Economic Growth Strategic Goals Strategic Objectives Strategic Objective 3.3: Inspire & Engage the Public in Aeronautics, Space, & Science Agency Priority Goals Office of STEM Engagement Cross-Agency External Multi-year Internal Multi-year Priority Goals Performance Goals Performance Goals External Annual Internal Annual Performance Performance Indicators Indicators Mission Directorates/Mission Support Offices Strategic Strategic Enhancement Multi-year Performance Performance Based on NSF Performance Goals Assessment Assessment Annual Performance Performance Questions Questions Indicators Assessment Model Success Criteria Success Criteria Draft 3-20-2018
Proposed Office of STEM Engagement Performance Assessment Strategy Alignment to the NASA Public & STEM Engagement Focus Areas NASA Public & STEM NASA Public & STEM NASA Public & STEM Engagement Focus Area 1 Engagement Focus Area 2 Engagement Focus Area 3 Create unique opportunities Strengthen public Build a diverse future STEM for students and the public understanding by enabling workforce by engaging to contribute to NASA’s STEM powerful connections to students in authentic work in exploration and NASA’s mission and work. learning experiences with Workforce discovery 1. Provide a broad NASA’s people, content, and 1. Enhance student Development spectrum of facilities. experiences through Institutional opportunities to attract 1. Provide learning NASA unique projects and engage the public. STEM Literate opportunities to attract Research and challenges. 2. Target a diverse set of and engage all students 2. Enable public Citizenry Capacity audiences to increase in STEM, including those participation in and public understanding of historically contributions to NASA’s NASA’s work and to underrepresented and work. Programmatic Strategic Investment Areas enhance appreciation of underserved. 3. Provide opportunities STEM. 2. Enhance capabilities of for students and the Cross-Cutting Operational Support Areas 3. Provide direct public institutions and support public to share ideas and access to NASA missions educators to position engage with NASA in and discoveries in an them to better serve addressing real-world Government Digital array of settings, students, including those problems. capitalizing on historically Efficiency Assets technology to extend our underrepresented and reach and impact. underserved. Evaluation & Aligned to OMB 3. Cultivate strategic and Administration Accountability partnerships with state- Cross-Agency based regional and Priorities federal public, academic and private institutions to magnify NASA’s efforts and investments. Draft 3-20-2018
Recommend
More recommend
