Science Simulations to Address Assessment Challenges Dawn Cameron| Minnesota Department of Education | Specialty and Technical Innovations Paul Katula | Maryland Department of Education | Scoring Specialist Kristen DiCerbo | Pearson | VP of Learning Research and Design
Overview • Minnesota’s history and some challenges for developing sims • Maryland’s efforts to use sims to assess complex academic standards • Pearson’s research with rich sims and difficult to measure constructs June 2019 Leading for educational excellence and equity, every day for every one. | education.state.mn.us 2
Minnesota Science Assessment Program 2008: Online-only science assessment • 50% Technology-enhanced (TE) items • 50% Multiple choice items 2011: Minnesota started developing simulations • 1 Operational and 3 Field Test Simulations per year • MC, TE and Task Response item types associated June 2019 Leading for educational excellence and equity, every day for every one. | education.state.mn.us 3
Task Response Stats Grade P-value pbis IRT parameter a IRT parameter b IRT parameter c 5 0.38 0.36 0.70 0.77 0.04 n=13 (range 0.20 to 0.68) (range 0.27 to 0.44) (range 0.38 to 0.96) (ranges -0.68 to 1.84) (range 0.02 to 0.11) 8 0.58 0.44 0.71 -0.21 0.04 n=11 (range 0.44 to 0.74) (range 0.3 to 0.52) (range 0.43 to 0.88) (ranges -1.12 to 0.48) (range 0.02 to 0.06) HS 0.41 0.38 0.67 0.59 0.04 n=7 (range 0.18 to 0.73) (range 0.32 to 0.58) (range 0.48 to 1.10) (ranges -1.06 to 1.58) (range 0.01 to 0.05) June 2019 Leading for educational excellence and equity, every day for every one. | education.state.mn.us 6
Simulation items • Includes both Task Response and more traditional TE and MC item types • More effectively measure critical thinking and problem solving • Open-ended, constructed response without the cost of hand scoring • Can decrease amount of text and language load for a student June 2019 Leading for educational excellence and equity, every day for every one. | education.state.mn.us 7
Prerequisite Knowledge NGSS Structure •Three dimensions, intertwined •Phenomena & storylines Choice Hotspot Drag & Drop Test Question Formats Inline Media Upload Extended Text •QTI Standard Interactions Simple Text Ordering •Portable Customized Interactions Matching Hot Text
Where We’re Headed PCI Type Input Output Score Non-standard Interaction Computer Animation Simulation Capture
Example from Our Science Practice Test Slightly better than a video All results pre-programmed Easy to convert for accommodations
Capture PCI Brainstorming Games • Exploration, Dissections • Based on a video game model, where players earn points for making moves. Trials • Quality Testing, Life Science • In conjunction with questions, determine if test-takers have collected sufficient evidence to give the answers. Random • Weather, Behavior • Where events can be introduced with random strength, frequency, timing, etc.
Scenario / Storyline Act 1 Phenomenon • Your high school’s football coach wants to know how to get the most distance on a field goal attempt in a real-game situation. Act 2 Background Research • How strong is the football team’s place kicker? What’s the initial velocity of the ball after it leaves his foot? Act 3 Investigation • Use a device that allows you to set the angle of a ball’s trajectory and the force with which the ball is “kicked.” Find the optimal angle.
Computations We Must Do Trigonometry Knowledge Can’t Be Assumed •Vectors require trigonometry •We can’t assume students would be able to perform trigonometric math on our science test. Assessment Boundary Can’t Be Exceeded •2D motion goes above the assessment limit •The assessment boundary in the NGSS PE being assessed specifically limits us to motion in one dimension. Evidence Statement 3.c (and 2.a.i) A typical football weighs 0.4 kg, and a kicker’s foot pushes it for 0.1 sec. Students express the relationship How much force is required for the football to leave his foot with a F net = ma in terms of causality, speed of 15 m/s? 30 m/s? What if the football had a mass of 0.5 kg? namely that a net force on an object causes the object to accelerate.
Level Setting the Simulation Parameters The simulation allows students to vary the following parameters: • Launch angle ( θ ) • Initial speed of the kick However, the simulation is reset after the preceding questions so that all students work with the correct force to get the launch velocity they set. Evidence Statement 1.a requires that students be able to organize data that show the net force, acceleration, and mass (held constant) of a macroscopic object. Rules allow footballs to weigh up to about 0.43 kg. Run the simulation to show the coach whether he should use a lighter or heavier football for his team to kick. Explain how any data points you obtained from the simulation support your advice to the coach.
Mastery of the S.E.P. is NECESSARY and SUFFICIENT. Height (m) vs. time (s) With each run, students are given graphs to describe the flight of the football and a data table is filled in, which students can sort or delete. Data must be used to answer the questions. h t Vertical Velocity (m/s) vs. time (s) v t Evidence statements 3.a and 3.b call on students to use the data as evidence, so the simulation allows them to select rows from the table (which they “generated” themselves) and drag them into a receiving bay for the questions they want to use those trials from their data to support.
Scoring In answering a question like the one shown earlier, for example (2.a.i): Rules allow footballs to weigh up to about 0.43 kg. Run the simulation to show the coach whether he should use a lighter or heavier football for his team to kick. Explain how any data points you obtained from the simulation support your advice to the coach. We would expect students to drag in rows that hold all variables constant (such as angle and initial speed) constant while varying the mass of the football. We would also expect to see a few trials at each mass and could score responses accordingly, based on how well the student: • Used the data to make valid and reliable scientific claims and determined an optimal solution (SEP) • Integrated empirical evidence to distinguish between cause and correlation (CCC)
Evidence from Science Simulations Pearson Kristen DiCerbo Jinnie Choi Matthew Ventura Emily Lai Minnesota Department of Education Dawn Cameron Jim Wood Judi Iverson June 2019 Image by XYZ
Problem to be Solved
How can we... Assess difficult to measure skills ● Make use of students’ process data ● Pull apart practices from knowledge for ● formative feedback
Approach
Evidence-centered design 23
Hard-to-measure standards Five simulations were designed to help Gravity Physics teach and assess Minnesota Science Standards. Physical Science Earth Science Additionally, we targeted to teach and The universe Motion, Energy assess the standards on the science 5.2.2.1.3 Force and motion 8.3.3.1.3 Describe how mass and practice, which involve investigation and 6.2.3.2.1 Kinetic-potential energy distance affect the gravitational force argument from evidence. transformation Satellite Submarine Biome The Nature of Science and Engineering The Nature of Life Science Interactions among science, technology, Science and Engineering Interdependence among living systems engineering, mathematics and society The practice of engineering 5.4.2.1.2 Impact of changes in parts 8.1.3.4.1 Use maps to describe patterns 6.1.2.2.1 Apply an engineering 7.4.2.1.1 Relationships among the parts in a and make prediction design process stable ecosystem 24
Simulation Templates 25
Evidence-centered design: We designed multiple tasks Each simulation contained multiple Gravity Physics activity/task variants. From a student’s interaction with a task, we Grade 5 Form: 6 task variants Grade 5 Form: 4 task variants collected multiple pieces of evidence based Grade 8 Form: 4 task variants Grade 8 Form: 4 task variants on the knowledge and skills we targeted in the design of the task. Satellite Submarine Biome Biome Biome One form, 6 task variants One form, 6 task variants One form, 5 task variants One form, 5 task variants One form, 5 task variants 26
A priori hypotheses about evidence model 27
Study participants For each simulation... We collected data from two groups of students- lower and upper grade levels. The sample sizes for the study only Total number of 5th grade 8th grade included the students who had interaction students data from the simulations as well as the matching near- and far-transfer scores. Gravity 316 193 123 Physics 198 104 94 Biome 304 156 148 Satellite 281 6 * 275 Submarine 253 89 164 28 MN Simulations Pilot Update |
Recommend
More recommend
