A Framework for Analyzing the Effects of Interruptions of Online - PowerPoint PPT Presentation

A Framework for Analyzing the Effects of Interruptions of Online Assessment Joseph Martineau Senior Associate June 22, 2016 1

Based on a Paper Prepared for CCSSO Available on the web from: – CCSSO – The Center for Assessment Martineau, J., Domaleski, C., Egan, K., Patelis, T., & Dadey, N. (2015, October). Recommendations for Addressing the Impact of Test Administration Interruptions and Irregularities. Washington, DC: Council of Chief State School Officers and Center for Assessment. June 22, 2016 2016 CCSSO NCSA Presentation on Test Interruptions 2

Why Plan for Interruptions? • Online test administration is incredibly complex, involving the interaction of… – Complex Test Registration and Deliver Applications (among others) – Capacity and Efficiency of Application, Database, and Web Servers – Varying capacity and quality of… • National, state, and local electrical grids • National, state, and local Internet backbones • Vendor, state, and regional agency/district/school internet service providers • District/school Ethernet and/or Wi-Fi infrastructure – Various brands and versions of… • Devices (e.g., brands and versions of desktops, laptops, thin clients, tablets) • Operating systems (e.g., versions of Windows, Linux, OS X, Chrome, Android, iOS) – Vendor, state, regional agency, district, and school personnel – A wide variety of personnel roles in both technical and administrative functions – Students – Nefarious internal and/or external actors June 22, 2016 2016 CCSSO NCSA Presentation on Test Interruptions 3

A Possibly Naïve Representation of the Degree of Complexity In Computer-Based Test Administration From Dubberly, H. (2002). Using Concept maps in product development: Preparing to redesign java.sun.com. Retrieved February 18, 2016 from content/uploads/2009/12/ddo_article_java_map.pdf. http://www.dubberly.com/wp- • But seriously, what could possibly go wrong? June 22, 2016 2016 CCSSO NCSA Presentation on Test Interruptions 4

Key Student-Level Causal Question • What would a student’s score have been if he or she had not been interrupted? – We don’t have an observable counterfactual – A causal modeling framework may be useful for analyzing the data, but the analyses do not produce causal effects • Three approaches (all involve matching) – Between-students comparison of those who experienced an interruption vs. those who did not (gives estimates of mean effects of interruptions) – Between-student comparison of students that experienced an interruption vs. those that did not controlling for degree of interruption (gives estimates of mean effects of varying degrees of interruptions) – Within-student comparison of achievement estimates before and after interruption (gives estimates of effects of interruptions on individual student scores). June 22, 2016 2016 CCSSO NCSA Presentation on Test Interruptions 5

Key Aggregate-Level Causal Question • What would a classroom/school/district aggregate score have been had – No students been interrupted during testing? – Completion of testing not been affected by other students’ interruptions? – Participation in testing not been affected by other students’ interruptions? • We don’t have observable counterfactuals for these questions – Question 1 can be analyzed using a pseudo-causal modeling framework, but the results do not produce a causal effect. – Questions 2 and 3 are more problematic • Our recommended approach – Flag aggregation units that show practically significant deviations from historical completion/participation rates – Note the number of students experiencing each type of interruption that are included in aggregate scores. June 22, 2016 2016 CCSSO NCSA Presentation on Test Interruptions 6

Framework to Analyze & Respond to Interruptions 1 2 Cleanse data and create data elements to Describe prevalence of each class represent instances of each class of of slowdown by demographic interruption and degree of test completion group and educational entity 3 Establish effects of slowdowns on aggregate completion rates 4A 4B 4C Estimate effects of Estimate effects of slowdowns Estimate effects of slowdowns slowdowns on aggregate on item scores on student scores entity scores 5A 5B 5C Make judgment whether Make judgment whether Make judgment whether to to invalidate or flag item to invalidate, flag, or adjust invalidate, flag, or adjust aggregate scores (or do nothing) student scores (or do nothing) entity scores (or do nothing) 6A 6B 6C Give cautions for use and Give cautions for use and Give cautions for use and interpretation of item scores interpretation on score interpretation on score reports and in score reports and data files reports and in data files for in data files for affected aggregate for affected item scores affected student scores entity scores Item Score Analyses Student Score Analyses Aggregate Entity Score Analyses June 22, 2016 2016 CCSSO NCSA Presentation on Test Interruptions 7

Framework to Analyze & Respond to Interruptions 1 2 Cleanse data and create data elements to Describe prevalence of each class represent instances of each class of of slowdown by demographic interruption and degree of test completion group and educational entity 3 Establish effects of slowdowns on aggregate completion rates 4A 4B 4C Estimate effects of Estimate effects of slowdowns Estimate effects of slowdowns slowdowns on aggregate on item scores on student scores entity scores 5A 5B 5C Make judgment whether Make judgment whether Make judgment whether to to invalidate or flag item to invalidate, flag, or adjust invalidate, flag, or adjust aggregate scores (or do nothing) student scores (or do nothing) entity scores (or do nothing) 6A 6B 6C Give cautions for use and Give cautions for use and Give cautions for use and interpretation of item scores interpretation on score interpretation on score reports and in score reports and data files reports and in data files for in data files for affected aggregate for affected item scores affected student scores entity scores Item Score Analyses Student Score Analyses Aggregate Entity Score Analyses June 22, 2016 2016 CCSSO NCSA Presentation on Test Interruptions 8

Framework to Analyze & Respond to Interruptions 1 2 Cleanse data and create data elements to Describe prevalence of each class represent instances of each class of of slowdown by demographic interruption and degree of test completion group and educational entity 3 Establish effects of slowdowns on aggregate completion rates 4A 4B 4C Estimate effects of Estimate effects of slowdowns Estimate effects of slowdowns slowdowns on aggregate on item scores on student scores entity scores 5A 5B 5C Make judgment whether Make judgment whether Make judgment whether to to invalidate or flag item to invalidate, flag, or adjust invalidate, flag, or adjust aggregate scores (or do nothing) student scores (or do nothing) entity scores (or do nothing) 6A 6B 6C Give cautions for use and Give cautions for use and Give cautions for use and interpretation of item scores interpretation on score interpretation on score reports and in score reports and data files reports and in data files for in data files for affected aggregate for affected item scores affected student scores entity scores Item Score Analyses Student Score Analyses Aggregate Entity Score Analyses June 22, 2016 2016 CCSSO NCSA Presentation on Test Interruptions 9

Framework to Analyze & Respond to Interruptions 1 2 Cleanse data and create data elements to Describe prevalence of each class represent instances of each class of of slowdown by demographic interruption and degree of test completion group and educational entity 3 Establish effects of slowdowns on aggregate completion rates 4A 4B 4C Estimate effects of Estimate effects of slowdowns Estimate effects of slowdowns slowdowns on aggregate on item scores on student scores entity scores 5A 5B 5C Make judgment whether Make judgment whether Make judgment whether to to invalidate or flag item to invalidate, flag, or adjust invalidate, flag, or adjust aggregate scores (or do nothing) student scores (or do nothing) entity scores (or do nothing) 6A 6B 6C Give cautions for use and Give cautions for use and Give cautions for use and interpretation of item scores interpretation on score interpretation on score reports and in score reports and data files reports and in data files for in data files for affected aggregate for affected item scores affected student scores entity scores Item Score Analyses Student Score Analyses Aggregate Entity Score Analyses June 22, 2016 2016 CCSSO NCSA Presentation on Test Interruptions 10