PARCC Diagnostic Assessments for Mathematics Comprehension: A - PowerPoint PPT Presentation

PARCC Diagnostic Assessments for Mathematics Comprehension: A Diagnostic Classification Model Approach Laine P. Bradshaw The University of Georgia June 24, 2015

Overview • A brief introduction to diagnostic classification models (DCMs) » How are they different from the models that are typically used in assessment? » Why are they particularly useful for diagnostic assessments? • How did we use the DCM framework to develop PARCC’s diagnostic assessment system for mathematics comprehension in Grades 2-8? • What new challenges in reporting exist when transitioning to a new psychometric framework?

Big Picture: Math Content to be Measured

Typical End of Grade Summative Test

Traditional Measurement Models for Summative Tests • Traditional testing procedures measure an overall ability in an area with a continuous latent variable Cluster 2 Item Cluster 4 Item Responses to items Math ability is 7 th grade Cluster 2 are observed continuous Item Math Ability Cluster 1 Item … Cluster 3 Item The more math ability a person has the more likely he or she is to answer an item correctly

Traditional Testing and Classification Methods 7 th Grade Math Ability LOW HIGH Not Proficient Cluster 2 Proficient Item • Information from Continuum: Cluster 6 7 th grade Item » Spencer has more math ability than Sue Math Cluster 1 Ability Item » Hugh scored a 240 on the test » Juan scored in the 70 th percentile … Cluster 3 • Diagnosis from Cut Score: Item » Sue scored below the cut score Sue is not proficient in 7 th grade math » Question difficult to answer: What are Spencer’s weaknesses?

Diagnostic Approach Instead of measuring an overall math ability in 7 th grade, we can break “math” down into a set of skills or attributes : Cluster 1 Cluster 2 Cluster 3 Cluster 4

Diagnostic Approach Cluster 2 Item Cluster 1 Cluster 4 Item Cluster 2 Cluster 2 Item Each cluster has two levels Cluster 1 Cluster 3 Item … Cluster 3 Cluster 4 Item A student can be in one of two groups, or levels, for each cluster • Higher group (masters; on-track) Lower group (non-masters; needs attention) • Masters are more likely than non-masters to answer items correctly.

Diagnostic Classification Models • DCMs uses responses to items to place students into groups according to multiple skills – No cut score is used to put students into groups; the model is built to do that Non-masters Masters Cluster 1 Add Subtract Cluster 2 Multiply Cluster 3 Divide Cluster 4

Diagnostic Classification Models • Students receive attribute-specific feedback Multiply Cluster 1 Cluster 2 Cluster 3 Cluster 4 Add Subtract Divide Spencer has mastered Cluster 1 and 2, but should improve his understanding of Cluster 3 and 4. .76 .84 .35 .28 • The model provides a probability each attribute is mastered • Notice there is no “score” in a traditional or grading sense

Benefits of Using DCMs • Aligns with a standards-based view of achievement where mastery of all standards is monitored » Modeling multiple “dimensions” or traits » Instead of answering: Did this student meet the standards? » Answer: Which standards has the student met? • DCMs provide valuable information with fewer data demands » Higher reliability per dimension than IRT/MIRT models » Potential to drastically reduce testing time » Quick and dirty categorical feedback

DCM Reliability 1 DCM 0.9 0.8 0.7 IRT Reliability 0.6 0.5 0.4 0.3 0.2 0.1 0 3 13 23 33 43 53 63 73 83 93 Number of Items 8 Item Reliability Templin, J., & Bradshaw, L. (2013). Measuring the reliability of diagnostic classification model examinee estimates. Journal of Classification , 30 (2), 251-275.

Psychometric Approach for Mathematics Comprehension Non-summative Assessments

Mathematics Comprehension

Example: Cluster Level Attributes in Math Number Systems Learning Progression Attribute 1: Divide Fractions Attribute 2: Understand Rational Numbers Attribute 3: Operate with Rational Numbers

Example: Cluster Level Attributes in Math Number Systems Learning Progression • Create a separate diagnostic test for each individual cluster Operate with Divide Understand Fractions Rational Rational Numbers Numbers … … …

Diagnostic Feedback to Students Example Example Example Student A Student B Student C 1 Number System 0.9 Mastery Probability 0.8 Results 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 Divide Fractions Diagnostics Understand Rational Numbers Operate with Rational Numbers Diagnostic On-track Needs Improvement Key:

Key to Flexible Diagnostic Assessment System: One Test Per Cluster, Packaged in Different Ways Individual Cluster Test 1 attribute ~8 items

Key to Flexible Diagnostic Assessment System: One Test Per Cluster, Packaged in Different Ways Grade Level Test 6 attributes ~48 items

Key to Flexible Diagnostic Assessment System: One Test Per Cluster, Packaged in Different Ways Learning Progression Test 6 attributes ~48 items

Key to Flexible Diagnostic Assessment System: One Test Per Cluster, Packaged in Different Ways Grade Level Test 6 attributes ~48 items Individual Cluster Test 1 attribute ~8 items Learning Progression Test 6 attributes ~48 items

Challenges in Reporting • DCMs do not provide a score! » They provide classifications • How will teachers and students interpret classifications? • How will they interpret the probabilities of mastery? • What is the best way to present the results so that interpretations are clear and accurate?

At the class level • Teachers can monitor the percentage of students who have mastered each cluster for a given grade level » Aggregate feedback

At the class level • Can show the multidimensional mastery profiles for each student in the class • Who has mastered each cluster?

At the student level • Should the probability of mastery be provided? Where? » Analogous to decisions for where to put standard errors on score reports for IRT-based assessments

Classification-based Reporting • DCMs are parametric, latent class models » Other examples of these models exist » For example, Bayes Nets • Models in the same family produce the same student estimates—mastery classifications and probabilities of mastery » These models face similar challenges in reporting

Please feel free to contact me with any questions or comments: laineb@uga.edu