A Computer Adaptive Alternate Assessment in Science Gary W. Phillips, Vice President & AIR Institute Fellow Paula Sable, Senior Test Developer American Institutes for Research June 22, 2016 2016 CCSSO/NCSA
Outline Outline 1. Alternate Assessment State Collaboratives 2. Advantages of a computer adaptive alternate assessment 3. Calibration 4. Ability estimation 5. Simulations for the Adaptive Algorithm 6. Test Design 7. Development 8. Content standards – Aligned with NGSS 9. Test Blueprint 2
1. Alternate Assessment State Collaborative
Stat State Collaborativ e Collaboratives There are three collaboratives for alternate assessments based on alternate achievement standards (AA-AAS) for students with the most significant cognitive disabilities. National Center and State Collaborative (NCSC) hosted by the 1. National Center on Education Outcomes (NCEO) at the University of Minnesota. Development was federally funded. Dynamic Learning Maps (DLM) hosted by the Center for 2. Educational Testing and Evaluation (CETE) at the University of Kansas. Development was federally funded. Multistate Assessment Collaborative (MAAC) hosted by the 3. American Institutes for Research (AIR) in Washington, DC involving 6 states. Development was state & AIR funded. 4
Multistat Multistate Alt Alternat rnate Assessment Collaborativ e Assessment Collaborative FT Designs State Peer -independent (first Review Vertical -operational Psychometric Year) Grades Approval Scale -embedded Growth Model NM S Gr. 4, 7, 11 Yes No IFT & EFT Within 1 parameter (2007) SS Gr. 11 Grade Partial Credit SC S Gr. 3-5, 6-8 Yes No IFT & EFT Within 1 parameter (2007) Bio Gr. 10 Grades Partial Credit SS Gr. 3-5, 6-8 DE R,M Gr. 2, 3-5, 6-8, 9-11 Submitted Yes R,M Within & 1 parameter (2011) S Gr. 5, 8, HS (10) IFT & EFT Across Partial Credit SS Gr. 4, 7, HS (9) S,SS Grades OFT & EFT HI R,M Gr. 3-5, 6-8, 11 Submitted Yes OFT & EFT Within & 1 parameter (2013) S Gr. 4, 8, 11 (CAT) Across Partial Credit Grades OH ELA, M Gr. 3-5, 6-8, 9-11 Submitted Yes OFT & EFT Within & 1 parameter (2013) S Gr. 5, 8, HS Across Partial Credit SS Gr. 4, 6, HS Grades WY ELA, M Gr. 3-5, 6-8, 9-11 Submitted Yes OP Within & 1 parameter (2014) S Gr. 4, 8, 9-11 Across Partial Credit Grades 5
2. Advantages of an Adaptive Alternate Assessment
Adva Ad vantages o of Adaptiv aptive Alt Alternat rnate Assessment e Assessment Aligned to Next Generation Science Standards (NGSS) Every student test meets the test blueprint Higher reliability of the scores for individual students because test difficulty is adapted to student ability Administered and scored by computer so the entire test and scoring takes about one hour Immediate score results Meets the same APA/AERA/NCME technical requirements as assessments of the general population Relatively inexpensive 7
3. Item Calibration
Calibration Calibration Partial Credit Rasch Model Within-Grade Scales for grades 4, 8 and 11 9
10 10
11 11
12 12
4. Proficiency Estimation
Student Ability Estimation Partial Credit Rasch Model Pattern Scoring 14 14
5. Simulations for the Adaptive Algorithm
Simulations Simulations HSA-Alt science tests are administered on an iPad as a CAT to students in grades 4, 8, and high school. The simulation was based on 5000 iterations for each test. Simulations were conducted to make sure that the CAT algorithm worked as expected. Three major outcomes investigated were: % Meeting blueprint % Item exposure rate Conditional standard error of measurement (CSEM) across the proficiency scale 16 16
Simulation R Simulation Results sults No blueprint mismatch was found in the simulations. The between student exposure rate was less than 0.2 for 43– 53% of the items and 0.4 for 57–76% of the items across the three tests. Given the limited pool size, it means that the majority of the items have relatively low exposure rates. CSEM s between the simulation result and the 2015 science operational administration. The red dots are from the simulation. The plots show that the CSEMs from simulations are relatively smaller than those from the 2015 administration, especially in the high school test. Indicates the CAT is more reliable than two-staged fixed forms (even though the pool in limited). 17 17
CSEM CSEM 18 18
CSEM CSEM 19 19
CSEM CSEM 20 20
6. T 6. Test Design est Design
Test Design f st Design for Alt or Alternat rnate CA e CAT T Assessment Assessment All HCPS Standards addressed at each grade (grades 4, 8, and 11 are assessed) o A total of 36 items is administered in grade 4. o A total of 40 items is administered in grade 8. o A total of 40 items is administered in gr. 11 o Minimum number of items established for each standard (varies by benchmark in each grade) o Maximum number of items established for each standard (varies by benchmark in each grade) o Items drawn from larger pool of items within the item bank 22 22
7. Development
De Development of Ha lopment of Hawaii AL ii ALT Assessment Standar T Assessment Standards s for Alt r Alternat rnate Assessment e Assessment Along with content-level educators in Hawaii, AIR/HIDOE Reviewed alternate assessment standards and extensions from other states Reviewed Hawaii Content and Performance Standards Determined the essence of each strand or domain within each standard. Extracted key ideas from each essence to create extended standards Content extensions have recently been revised and are now called content specifications.
8. Content Standards
Connecting Ha Connecting Hawaii AL ii ALT Assessment T Assessment Standar Standards t s to Ne Next Generation Science xt Generation Science Standards Standar AIR science staff, in conjunction with HIDOE Created table to note where existing HCPS intersect with NGSS Using this bridge document, identified Alternate Assessment Content Specifications that could be used as applications of these connections. Identified existing alternate assessment items that reflect these connections. Created additional alternate assessment items to add to the pool of NGSS-related items 26 26
Alignment of Hawaii Content and Performance Standards to Next Generation Science Standards Examples Grade HCPS III HCPS III HCPS III NGSS NGSS Core AIR Recommended NGSS Related HCPS III Strand Standard Benchmark Grade/ Idea Performance Expectation Aligned Content description Discipline Specifications Application Physical, Standard 8: SC.8.8.2 Illustrate Middle Earth’s MS-ESS2 Develop a model Match the type of rock 8 Earth, and Physical, the rock cycle School to describe the cycling of to how it was formed. Systems Space Earth, and and explain how Earth and Earth’s materials and the Sciences Space igneous, Space flow of energy that drives Identify a type of rock. Sciences: metamorphic, Science this process. EARTH and and sedimentary Identify differences in SPACE rocks are rocks (e.g., size, SCIENCE: formed. texture). Understand Earth and its processes, the solar system, and the universe and its contents. 27 27
Alignment of Hawaii Content and Performance Standards to Next Generation Science Standards Examples Grade HCPS III HCPS III HCPS III NGSS NGSS Core AIR Recommended NGSS Related HCPS III Strand Standard Benchmark Grade/ Idea Performance Expectation Aligned Content description Discipline Specifications Application The Standard 2: SC.8.2.2 Describe Middle Earth’s MS-ESS2 Develop a model Identify how scale or 8 Scientific Scientific how scale and School to describe the cycling of mathematical models Systems Process Process: mathematical Earth and Earth’s materials and the can be used to support NATURE of models can be Space flow of energy that drives and explain scientific SCIENCE: used to support Science this process. data. Understand and explain that scientific data. science, technology, and society are interrelated . 28 28
9. Test Blueprints
Meeting the T Meeting the Test Blueprint st Blueprint Even though each student gets a different set of items, the items have been selected so that each set of tasks cover the state’s blueprint and content standards. 30 30
Meeting the T Meeting the Test Blueprint st Blueprint Standards addressed at Grades 4 and 8: The Scientific Processes Life and Environmental Sciences Physical, Earth, and Space Science 31 31
Meeting the T Meeting the Test Blueprint st Blueprint Standards addressed at High School: The Scientific Processes Life and Environmental Sciences Organisms and the Environment Structure and Function of Organisms Diversity, Genetics, and Evolution 32 32
Meeting the Test Blueprint Meeting the T st Blueprint For each Standard, there is a block with a fixed number of items Example: 11 items for Gr. 11 Scientific Process Within each block, there is a minimum and maximum number of items per benchmark 33 33
Recommend
More recommend
