Measuring What Matters with Innovative Assessments of 21 st Century Skills for College and Career Readiness: NAEP SAIL Virtual World for Assessing Online Inquiry in ELA Jesse R. Sparks Educational Testing Service CCSSO National Conference on Student Assessment June 25, 2019
NAEP SAIL Initiative The Survey Assessment Innovations Laboratory (SAIL) is a hothouse dedicated to explorations in the cognitive sciences, assessment, and technology that will enable a continuously updated NAEP which leads the field of educational assessment How we score and What we How we assess assess report what we assess New models of New technologies to elicit Computational advances for 21st century skills and capture complex automated scoring, complex human interactions and competencies modeling, score reporting 2
Motivation • There is a growing interest in teaching and learning of complex cognitive constructs: • Conducting inquiry from multiple sources in the English Language Arts • There is a need to develop quality large-scale assessments that provide valid information about students’ knowledge, skills, and abilities as related to authentic, complex inquiry tasks. 3
NAEP SAIL Project Aims • Research and develop virtual environments with authentic contexts • To gather evidence of students’ inquiry skills using naturalistic simulations reflecting real -world applications of the targeted skills • Explore students’ interactions in simulated inquiry scenarios that require • Locating, evaluating, reading , and writing syntheses from multiple sources, and reconciling conflicting, unreliable, and inaccurate information • Leverage technology from games and simulations • to enhance student motivation and engagement in assessment tasks (important for NAEP, which is “low stakes” for the students) • • to provide evidence of students’ cognitive processes as they unfold over time. 4
SAIL Virtual World for ELA: Environment & Task Design
Construct: Online Multiple-Source Inquiry 6
SAIL ELA Virtual World Environment 7
SAIL ELA Virtual World Environment: Tools and Resources Planning Inquiry with Virtual Partner (Guide) Compose Written Response to Inquiry Task Supports for Source Evaluation & Note-taking using Collected Resources for Saved Resources Simulated Questioning of Virtual Characters (experts or laypeople) Simulated Web Search & Library Search Tools to retrieve documents 8
SAIL ELA Virtual World – Scenario-Based Task Logic Scenario-based tasks in the ELA Virtual World include three phases: • Setup: a fixed sequence of introductory material and tutorials for each tool • Free Roam: a period of “free exploration” of available resources (ideal & optional) • Conclusion: a culminating phase where students work to integrate information and compose a final, written response to the inquiry question, using information from the available sources. 9
SAIL ELA Student/Evidence Model Alignment Each aspect of the SAIL ELA Virtual World is designed to measure a specific aspect of the online inquiry construct. All student actions are mapped to a specific dimension, and are evaluated with specific scoring rules. In ECD parlance, this represents our Evidence Model . Relations between Online Inquiry Construct & SAIL ELA Virtual World digital tools (white boxes): Define Problem Evaluate Sources; Example Communicate and Information Locate Sources Process, Analyze Results Evidence Model Needs & Synthesize Statements Use Communicator to Use Partner to Use Inspector to plan a response (e.g., In Evidence Manager define research examine sources identify intended Tool , student evaluates Use Evidence questions recipients) the usefulness of sources Manager to support for one’s inquiry task. close reading and analysis of sources Scored by number of Use Partner to Use Internet Use Communicator to usefulness ratings of 4 or 5 define known, Search to compose a response to of 5 for sources containing needed key information for retrieve websites the task information completing the task Use Map Tool to Use Librarian to Use Evidence Manager make plans for Use Evidence In Communicator Tool, retrieve print and Communicator to information Manager to support student includes citations sources support a response with gathering critical evaluation of of relevant and reliable evidence drawn from sources sources in their final sources (i.e., citations, response to the task. Use Conversations excerpts) Scored by number of key with Experts or references cited (0-3) Laypeople to retrieve information 10
SAIL ELA Scenario-Based Task & Structure • Goal : Evaluate historical accuracy of claims contained in artifact — Should it go in museum? • Three phases: Setup, Free Roam, and Conclusion. • Free Roam allows for nonlinear, free exploration and investigation of inquiry process . Phase Construct(s) Score Components Points (99 total) Define Identify best first step with Partner ; 13 Setup: Identify key information that must 1 best first step problem/info Planning be investigated; Identify 2 useful 3 claims to examine needs, Locate, 2 useful locations locations; Questioning an authority Inquiry Evaluate 1 question to authority figure figure; Evaluations in Evidence 6 Evidence Manager evaluations Manager tutorial Locate, Evaluate, Navigation; Search behaviors in 55 Free Roam: Internet Café & Library ; Questioning Process/Analyze/ 9 Library (2 search, 1 save, 6 EM Gather and behaviors in Faculty Offices ; Evaluations) Synthesize Resources viewed/saved to 20 Faculty Offices (6 questions, 2 Evaluate save, 12 EM Evaluations) Evidence Manager ; Evaluations in Resources 26 Internet Café (2 search, 3 Evidence Manager view, 3 save, 18 EM evaluations) True/false judgments, Source Conclusion: Evaluate, 26 attributions, Misinformation Process/Analyze/ 10 true/false judgments Integration corrections 10 source attributions Synthesize 6 corrections Task Final decision about inquiry task; Conclusion: Process/Analyze/ 5 [ 11 ] Argument writing; Use of excerpts; Synthesize, 1 final decision Argument Use of citations 3 citations Communicate 1 excerpt Writing Task [6 argument quality] 11
SAIL Virtual World for ELA: Empirical Work
SAIL ELA Empirical Work 2015-2017 • Phase 1: Developing the Virtual World Platform (N=14) • Play testing – Groups A&B – Storyboard (n=5) • Play testing – Group C – Playable Platform (n=5) • Play testing – Group D – Playable Platform (n=4) • Phase 2: Developing the Scenario-Based Task (N=31) • Play testing – Group E – Playable Task (n=6) • Play testing – Group F – Playable Task (n=5) • Play testing – Group G – Playable Task (n=5) • Cognitive Lab – Group H – Complete Task w/Think-Alouds (n=15) • Small-Scale Tryouts: Group SE – Complete Task in Classroom Conditions (n=134) 13
Coglab Data – Time per Scene/Tool Letter Evaluation Setup Phase Free Roam Writing 14
Key Findings from Empirical Work • Students demonstrated a variety of skilled performances during the virtual world inquiry task — multiple paths to solution were enabled and pursued by students. • Task scores correlated with external measures of ELA proficiency, including class grades and state test scores (e.g., PARCC, ACT Aspire) • Students struggled with information locating/search tasks and critical evaluation activities, including difficulty correcting inaccurate information. • However, students’ ability to correct inaccurate information predicted their performance on the final essay task. • Students found the virtual world task to be engaging, despite some tedious or time-consuming aspects. 15
Extending the SAIL ELA Virtual World to Assess Collaborative Inquiry Students Collaborating on ELA Inquiry Task in “Face to Face” Context In 2016, we began development of a collaborative version of the SAIL ELA Virtual World task, in order to elicit evidence of students’ collaborative skills, and to elicit better evidence of students’ online inquiry skills VIA collaborative dialogue. • Conducted “face to face” playtesting Digital Platform Enables Structured Collaboration in “Remote” Context with pairs of students to refine collaborative prompts • Conducted “remote” playtesting using a digital platform for collaboration , which enables delivery of collaborative prompts , real-time audio/video and text chat, and a shared task interface. 16
These Virtual Environments Enable Us To… • Deliver innovative assessment tasks using digital platforms • Engage students in simulated versions of authentic real-world tasks • Collect moment-by- moment data on students’ actions • Provide multiple paths to correct solutions, and opportunities to make students’ thinking visible • Better understand how students’ solution processes contribute to final outcomes 17
Recommend
More recommend
