Test Security: Its Different for State, Interim, and Classroom - PowerPoint PPT Presentation

Test Security: It’s Different for State, Interim, and Classroom Assessment Steve Ferrara Measured Progress June 27, 2018 In S. Valenzuela (Organizer), What’s Yours is Mine and What’s Mine is Yours , presentation in the National Conference on Student Assessment, San Diego

Overview  Address Sally’s requirements  Best practices: standardization, monitoring, forensics, investigations  Also: Resolution  Background  PDIR framework  Are all shared item banks equal? 2

PDIR framework  Prevention, Detection, Investigation, Resolution  Goals of test security  No security breaches  Completely resolve breaches that do occur  Evidence for the validity argument for intended score interpretations and uses (SIUs)  Support the data integrity claim so that you can support those SIUs Ferrara (2017) 3

PDIR framework  How are we (state assessment programs) doing on PDIR?  Some states are at risk  States and districts not well equipped to carry out effective investigations  Resolution may be spotty (we just don’t know for sure) Ferrara (2017) 4

Shared item banks  State accountability assessments  E.g., PARCC, Smarter, ELPA 21, WIDA, MSAA  A primary intended SIU: accounting for percentages of students at/above Proficient  Commercial interim assessment products  E.g., MAP, iReady, eMPower  A primary SIU: achievement growth toward a goal  Classroom assessment item banks and test assembly support  A primary intended SIU: student mastery of instructional content 5

Shared item banks  Advantages  Economic and operational  Risks  A security weakness in one state or locale is a breach for everyone  A breach in one state or locale is a breach for everyone  Consequences of security breaches  State accountability ***validity of SIUs, loss of assets  Interim *** validity of SIUs, loss of assets  Classroom assessment IBs *** validity of SIUs, loss of assets 6

Best practices: State accountability assessments  We organize our practices around the PDIR framework…  Within the limits of the RFP specifications What Can States Do?  Identify the highest risks for a specific program  Decide how best to mitigate those risks  Implement and enforce the elements in the PDIR framework 7

Best practices: State accountability assessments  Risk  Biggest threats appear to be when print materials are in schools and during administration for both print and CBT  Exposure of content, help to examinees  Mitigation—by the state  Training  Forensics, web patrolling  Monitoring (e.g., peer review citations) 8

Best practices: Commercial interim  Risk  Probably the same as state accountability, but with less intensity  But the cost of losing secure content is just as high  And the impact of undermining validity of score interpretations is higher (on students)  Mitigation—by the vendor  Training  Forensics, web patrolling  Monitoring? 9

Best practices: Classroom assessment item banks  Risk  Content is always available, discussing items with other teachers and students is part of the teaching-learning process  Is item exposure in one classroom, school, etc. a problem for other classrooms, etc.?  Mitigation  Training in effective classroom assessment practices  Other, TBD  (Sorry) 10

Best practices: Detection  Accountability, interim, classroom  DIY or contracted services  Your psychometrics vendor  Any test security companies that may exist  Lots of research on detection statistics  Multiple statistical approaches for the same security risk  Some holes (e.g., constructed responses, next generation assessments) 11

Best practices: Investigation  Accountability, interim, classroom  Educators are not trained to conduct effective, professional investigations  To the same degree that professional investigators are not trained as educators  At least for big, scary situations  Would a commercial interim provider investigate a suspected security breach?  A commercial classroom provider? 12

Best practices: Resolution  Accountability, interim, classroom  Needs  Policies so that everyone knows the rules and consequences  Support from Department, Board, Legislature  Courage and will 13

Reference Ferrara, S. (2017). A comprehensive framework for policies and practices to improve test security programs: Prevention, detection, investigation, and resolution (PDIR ). Educational Measurement: Issues and Practice , 36 (3), 5-23. 14

It’s all about student learning. Period.

Thank you. ferrara.steve@measuredprogress.org +1 603-749-9102, ext. 7065