Improving Data Quality in the Survey of Graduate Students and - PowerPoint PPT Presentation

Improving Data Quality in the Survey of Graduate Students and Postdoctorates in Science and Engineering (GSS) Peter Einaudi, RTI International Jonathan Gordon, RTI International Stephanie Eckman, RTI International Herschel Sanders, RTI International Mike Yamaner, National Science Foundation www.rti.org RTI International is a registered trademark and a trade name of Research Triangle Institute.

What is the GSS?  Establishment survey of all  Survey of Graduate Students U.S. academic institutions and Postdoctorates in Science that grant graduate degrees and Engineering (GSS) in science, engineering, and health (SEH) fields – Sponsored by the National Center for Science and Engineering  Institutions report enrollment Statistics (NCSES) within the and financial support data for National Science Foundation graduate students, postdoctoral (NSF) and the National Institutes researchers, and doctorate- of Health (NIH) holding nonfaculty researchers (NFRs)

Sample data grid  Voluntary survey  High response rates – 98.6% RR3 in 2016  Burdensome – 56.4 hours average  Electronic Data Interchange (EDI) limited prior to 2016

 Improving the value of the data – Users want to analyze master’s and doctoral data separately  Aligning taxonomy of disciplines across federal surveys Key Challenges  Ensuring all eligible units are reported and classified appropriately  Mitigating response burden and maintaining high response rates

Collect  Separate master’s and doctoral more enrollment data data  Increase use of EDI Goals of Reduce  Simplify coding of academic units Burden Redesign  Eliminate data entry errors Improve  Align request with administrative data Quality  Improve coding of disciplines

Proposed Solutions  Simplify and expand use of EDI  Replace GSS codes with Classification of Instructional – Align data request with Programs (CIP) respondents’ administrative data systems – Already utilized and maintained – Reduce reliance on manual by institutions for mandatory data entry reporting – Facilitate separate master’s – Improved comparability with and doctoral reporting without other federal surveys added burden – More granular coding scheme – Shifts burden of taxonomy changes from data reporters to data collectors

Fall 2015– Fall 2016– Fall 2017– Summer 2016 Summer 2017 Summer 2018 Institutional site visits Pilot Full implementation Recordkeeping and data Feasibility reporting practices Estimate reporting Coordinator survey burden Understand impact of changes

Fall 2015– Spring 2016– Fall 2017– Summer 2016 Summer 2017 Summer 2018 Institutional site visits Pilot Full implementation Recordkeeping and data Feasibility reporting practices Estimate reporting Coordinator survey burden Understand impact of changes

What we learned  Coordinators can distinguish  Most coordinators willing to between students enrolled use EDI in master’s and doctoral – Concerns about programming programs in their records effort required – Privacy concerns about transmitting individual-level data  CIP codes available in student information systems – Codes less available to report postdocs

Integrating lessons learned into the Redesign  Flexibility in field coding method  Allay concerns about privacy – CIP codes strongly encouraged for – Avoid Personally Identifiable student data Information – GSS or CIP codes for – Option to aggregate data postdoctoral data before transmission  Make EDI as simple as possible – Excel templates – Eliminate additional programming steps to format data

Fall 2015– Spring 2016– Fall 2017– Summer 2016 Summer 2017 Summer 2018 Institutional site visits Pilot Full implementation Recordkeeping and data Feasibility reporting practices Estimate reporting Coordinator survey burden Understand impact of changes

Trial Run: Pilot Data Collection GSS 2016  Ran parallel to GSS 2016 survey cycle – Stratified random sample of – Data request included: coordinators  Separate reporting of master’s and doctoral students  Master’s-only institutions ( n = 15)  Use of CIP codes to report student data  Doctorate-granting institutions  Use of EDI to transmit data with 15 or fewer units ( n = 25)  Doctorate-granting institutions with over 15 units ( n = 40)

Pilot Results  98.7% of coordinators  Nearly 90% of schools able to upload at least reported student data some data with CIP codes  All schools with both  Most coordinators reported master’s and doctoral similar or lower response students able to report burden compared to the these data separately previous year

Average response burden for pilot coordinators

Fall 2015 – Spring 2016 – Fall 2017 – Summer 2016 Summer 2017 Summer 2018 Institutional site visits Pilot Full implementation Recordkeeping and data Feasibility reporting practices Estimate reporting Coordinator survey burden Understand impact of changes

Preparation for full implementation  Targeted Communication  Training – Mail – Webinars – E-mail – Training videos – Conference presentations – Sandbox  Redesign of Survey Website – Highlight changes – Taxonomy tool

2017 Response method

Average response burden

Impact on response  Response rates declined… but more data – Increase in school-level nonresponse  Item nonresponse declined – Despite increase in items  Lower item nonresponse with EDI

Item nonresponse rates by section and response method

Discussion  Understand your  Pilot your changes respondents and their data systems  Communicate early and often  Design around constraints: one size does not fit all

Thank You peinaudi@rti.org Peter Einaudi jgordon@rti.org Jonathan Gordon myamaner@nsf.gov Mike Yamaner