Virginia Modified Achievement Standards Test (VMAST): Lessons Learned Shelley Loving-Ryder Virginia Department of Education June 20, 2011 2011 National Conference on Student Assessment Orlando, Florida
VMAST: Overview • GSEG to develop assessment for Grade 8 reading and mathematics • Adds supports and simplifications recommended by educators to existing online test items 2011 National Conference on Student Assessment Orlando, Florida
Activities to Date • Conducted focus groups and online surveys with teachers and special education directors on learning characteristics and support/ simplifications used with identified students in instruction • In fall 2008 and spring 2009 conducted 2 pilots using cognitive interviews with students from urban and rural school districts who were identified as likely VMAST participants 2011 National Conference on Student Assessment - 3 - Orlando, Florida
Field Test • Field test with approximately 500 students in May-June, 2010 for VMAST grade 8 reading and mathematics. • Students participating in field test were students with disabilities who met the participation criteria for VMAST • 47 field test students were administered unsupported items first and were then administered the supported items 2011 National Conference on Student Assessment Orlando, Florida
Results: Reliability Alpha Alpha Subject N-Count VMAST SOL Math 235 0.85 0.89 - 0.93 Reading 257 0.84 0.88 – 0.90 2011 National Conference on Student Assessment Orlando, Florida
Results of the Mathematics Field Test 2011 National Conference on Student Assessment Orlando, Florida
Math Supports and Simplifications • Provided hint boxes with formulas, strategies, and reminders, • Color coded important information • Provided online manipulatives • Simplified numbers • Simplified language in stems • Added or altered graphics • Reduced answer options from 4 to 3 2011 National Conference on Student Assessment Orlando, Florida
Overall Comparison of VMAST vs SOL for Grade 8 Mathematics VMAST is 40 MC items (vs 50 on the SOL test) measuring: • RC1: Number and Number Sense ; • RC2: Computation and Estimation ; • RC3: Measurement and Geometry ; • RC4: Probability and Statistics ; and • RC5: Patterns, Functions and Algebra . Average Test N Percent Std Dev Min Max Correct VMAST 47 59 17.52 8 95 SOL 47 48 15.28 20 80 2011 National Conference on Student Assessment Orlando, Florida
Mathematics Item-Level Analyses • Change in p-values for supported items • 5 items had an increase in p-value from .20 to .34 • 21 items had an increase in p-value from.10 to .19 • 8 items either had no change in p-value or the increase was less than .10 • 6 items had a decrease in p-value 2011 National Conference on Student Assessment Orlando, Florida
Item-Level Analyses: Overall Results • P-Value • Ranged from .34 to .90 • Mean of .65 • Point biserial • Ranged from .11 to .59 2011 National Conference on Student Assessment Orlando, Florida
Recap of Mathematics Results • Reliability and Validity look good • Students have higher percent correct on VMAST than on SOL • No atypical patterns in gender/ethnicity results • Nice range in item difficulty and item discrimination • Some supported items became more difficult • Consider number of supports added • Were supports used by students? 2011 National Conference on Student Assessment Orlando, Florida
Results of the Reading Field Test 2011 National Conference on Student Assessment Orlando, Florida
Reading Supports and Simplifications • Shortened reading passages • Excerpted relevant sections of reading passages and presented them with item • Provided hint boxes with reminder to return to passage when item dealt with passage as whole • Used graphic organizers to present passage information • Simplified language in stems • Highlighted key words • Reduced answer options from 4 to 3 2011 National Conference on Student Assessment Orlando, Florida
Overall Comparison of VMAST vs SOL for Grade 8 Reading VMAST is 36 MC items (vs 45 on the SOL test) measuring: • RC1: Use word analysis strategies and information resources ; and • RC2: Demonstrate comprehension of printed materials. Average Test N Percent Std Dev Min Max Correct VMAST 47 60 17.93 22 94 SOL 47 44 16.76 11 84 2011 National Conference on Student Assessment Orlando, Florida
Item-Level Analyses: Overall Results • P-Value • Ranged from .29 to .90 • Mean of .62 • Point Biserial • Ranged from .22 to .57 2011 National Conference on Student Assessment Orlando, Florida
Reading Item-Level Analyses • Impact of supports and simplifications on reading VMAST items – 10 items had an increase in p-value from .20-.38 – 16 items had an increase in p-value from .10-.19 – 10 items had p-values that were either unchanged or had an increase of less than .10 2011 National Conference on Student Assessment Orlando, Florida
Recap of Reading Results • Reliability and Validity look good • Students have higher percent correct on VMAST than on SOL • No atypical patterns in gender/ethnicity results • Nice range in item difficulty and item discrimination • Passage type may play a role in effectiveness of the supports and simplifications 2011 National Conference on Student Assessment Orlando, Florida
Lessons Learned: Effective Supports • Hint boxes • Simplifying language • Simplifying presentation of item • Drag and drop tool to fill in equations or expressions • Color coding when used judiciously • Highlighting • Excerpted text • Reminder to return to passage when item addresses passage as a whole 2011 National Conference on Student Assessment Orlando, Florida
Lessons Learned Supports are most helpful when they are consistently displayed across items • Hint boxes always in the same location • Components that can be dragged and dropped identified consistently • Same “icon” to activate animation 2011 National Conference on Student Assessment - 19 - Orlando, Florida
Lessons Learned • Must be judicious in use of supports: “oversupporting” can be confusing to students • Supports and simplifications not helpful if students don’t know content • Providing practice with items is crucial. Practice items and guides for teachers may be found at www.doe.virginia.gov/testing/alternative_assessments/ vmast_va_mod_achievement_stds_test/practice_items/ index.shtml 2011 National Conference on Student Assessment Orlando, Florida
Next Steps • Moving toward implementation of VMAST in reading and mathematics in grades 3-8, Algebra I and end-of-course reading • Field tested mathematics in spring 2011 • First operational administration of mathematics in spring 2012 • Field test reading in spring 2012 • First operational administration of reading in spring 2013 2011 National Conference on Student Assessment Orlando, Florida
Application to Next Generation Assessments • Continue to study the characteristics of students to ensure that participation criteria are clear and appropriate • Use principles of universal design in item development • Reduce language load • Simplify presentation of items • Maximize the use of technology to develop additional supports and simplifications 2011 National Conference on Student Assessment Orlando, Florida
Contact Information: Shelley Loving-Ryder Virginia Department of Education student_assessment@doe.virginia.gov (804) 225-2102 2011 National Conference on Student Assessment Orlando, Florida
Recommend
More recommend
