The Im Impact of In Increased Teacher Assessment Lit iteracy & Use of Formative Feedback Year-2 Study
Cast of Today’s Characters • Michael R. McCormick, Superintendent Val Verde USD • Jennifer M. Doskocil, Elementary Coordinator • Sandy Sanford, EdD, Assessment Consultant • Pete Goldschmidt, PhD, Research Consultant
Short History of Val Verde Elementary Benchmarks
2002-2013 • 95% Multiple Choice Pre-Year Benchmark Analysis & Action Benchmark 1 Analysis & Action • ~75% of Standards Benchmark 2 School Year Analysis & Action • 20 to 40 items Benchmark 3 Analysis & Action • 85% Scan Sheet Post-Year Benchmark Analysis & Action
“formative analysis” & Action 2002 -2013 • By Grade Levels Teams • Reduction from Standard to Items • Using EADMS Analytic Tools • Isolate Offending Standard/Items(s) o e.g., Wrong Answer Analysis • Determine Cause(s) • Determine Instructional Fix • Apply Instructional Fix • Intensely Collaborative (at least initially)
Downside of 2002-2013 • Lockstep System • District Dictated — Not Teacher Owned • Competition Increased as years passed • Collaboration Decreased as years passed • Original Purpose Compromised • Ignored the “Formative Assessment” Revolution • Could not effectively deal with new SA item types and PTs
---Collaboration Capital--- Build your SHARING REPUTATIO -Michael R. McCormick N Val Verde USD
Influence of the New “Summative Assessment” Characteristic OLD in Math NEW in Math Administration Paper & Pencil Computer Test Components One MC CAT & PT MC, MS, EQ , TM, TI, DD, GR, ST Item Types MC Responses One Correct per Item May have Many Correct per Item Cognitively Complexity Lower Higher Psychomotor Dependent Little Lots Results Math beat ELA ELA beat Math
MC Question Are things like the required use of a Computer & Technology Enhanced Items (TEIs) … A. needless Contaminants creating needless barriers? (Construct Irrelevant Variance) B. essential Components for CCR in the 21 st century?
Pilot Study 2015-16
Our Need at time of Pilot Transition to an assessment/analysis approach that … • minimizes Competition and maximizes Collaboration • features a Teacher-Driven approach to assessment/analysis as opposed to a District-Dictated approach • incorporates the instructional use of Formative Interactions • considers the psychomotor aspects of Computer use & TEIs
A Word about the “F” word “formative analysis” = analyzing any assessment in order to act formatively “Formative I nteraction” = using the assessment to interact in a formative manner with students (e.g., complex interactions) “FORMATIVE Assessment” = assessment at the point of instruction and incorporated systematically in the instructional process.
Pilot Research Questions (2015-2016) • Will teachers given full authority to choose all assessment items (Control Group) OR teachers choosing half the assessment items with an expert choosing the other half (Treatment Group) build assessments that align better to Smarter Balanced specifications with respect to mix of DOK, Item Types, Task Models, & Claims? • Which Group will perform better on the Summative Assessment?
Pilot Structure Characteristic 2015-16 5 th Grade Grade Level Content Area Math Treatment Group 6 Sites (Random) Treatment Assignment 8 Item Testlet Testlet Unique to Each Teacher (~27) Items Selected by Teachers (4 Items) & Expert (4 Items) Control Group 6 Sites (Random) Control Assignment 8 Item Testlet Testlet Unique to Each Teacher (~27) Items Selected by Teachers (all 8 Items) Assignment Iterations 5 Iteration Interval Approx. 6 weeks Administration Window ~2 weeks at end of instruction
Item Alignment • Each item created to conform to the appropriate Smarter Balanced Item Specification Task Model • Testlet Mix (Item Type, Task Model, DOK, & Claim) as close as possible to SA variety as described in Blueprint and Item Specifications
Claim 1 Target C
Experiment — Pilot (Candidate List) • Expert builds Candidate List (CL) of 20 items in advance of each Assessment Cycle. Items are authored in EADMS • Based on 2-3 Priority Standards per Cycle • Mix of Item Types, Task Model, DOK, & Claim aligned to Summative Assessment • CL cover sheet (Group A or B specific) describing process with the following sheets showing each item with indication of standard, item type, Task Model, DOK, and Claim/Target • Items not used for Testlet could be used by teachers to support lessons
Pilot Testlet System Group A Choose 8 Items Teachers Candidate Item List Published Study List Group B Choose 4 Item Group B Testlets Created in 4 More Items Added EADMS Testlets Numbers Teachers Admin Scores Captured in Sent to Teachers Testlets EADMS Results Analyzed Results Harvested Administration W/R Summative Assessment from EADMS Monitored in EADMS
Professional Development — Pilot • 3.5 hours Up-Front, On-Site PD re the value of item selection o Building Testlet o Measuring purposely o Standard, DOK, Item Type, Task Model mix, Claim mix o Formative Interaction o TEI psychomotor alert • Optional “On Demand” extended PD • Continual emphasis via Instructional Coaches • Reminders & emphasis via Candidate Item Lists with each Assessment Cycle
Role of the Testlet in Pilot • “formative Tool” o Short-Cycle Interim (Benchmark) o To be ”formatively analyzed” and acted on accordingly • Remaining CL items could be used to support instruction
Pilot Study Results • No significant difference in Treatment and Control Groups w/r SA • Testlet Quality converged (required PD same for all) • Teachers still used Testlets more Summatively than Formatively • Teachers wanted Testlet symmetry across grade level at each site • Psychomotor challenges re Computer & TEIs greater than thought
Year 2 Study 2016-17
Our Need for Year 2 Study Improving the assessment/analysis approach so as to … • minimize Competition and maximizes Collaboration • feature a Teacher-Driven approach to assessment as opposed to a District-Dictated approach • Move to school (grade level) based Testlets • Accelerate Formative Interaction towards FORMATIVE Assessment • Accelerate help for psychomotor aspects of computer & TEI
Year 2 Research Questions Analysis of Fidelity of Implementation What is the … • 1. relationship between Professional Learning & Testlet Quality? • 2. relationship between Professional Learning & SA Outcomes? • 3. relationship between Testlet Quality & SA Outcomes?
Pilot & Year 2 Structure Characteristic 2015-16 2016-17 5 th Grade 3 rd , 4 th , & 5 th Grades Grade Level Content Area Math Math Treatment Group 6 Sites (Random) 12 Sites Treatment Assignment 8 Item Testlet 8 Item Testlet Testlet Unique to Each Teacher (27) Each Site (12) Items Selected by Teachers (4 Items) & Expert (4 Items) Grade Team, Expert, or Both Control 6 Sites (Random) Control Assignment 8 Item Testlet Testlet Unique to Each Teacher (27) Items Selected by Teachers (all 8 Items) Assignment Iterations 5 9 Iteration Interval Approx. 6 weeks Approx. 4 weeks Administration Window ~2 weeks ~4 weeks
Year-2 Indicators • Proximal = Testlet Quality • Distal = Summative Assessment Outcomes
Mediation Study formative Tool Summative Professional Assessment Development Results
Year 2 PD • Continuation of Pilot PD • More explicit Direction of Pilot PD • More detail on Formative Techniques • Provision and Explanation of Formative Tools (e.g., Low Tech to High Tech Process and Power Points) • Greater emphasis on Computer & TEI re psychomotor implications • More On-Demand On-Site PD
Year 2 Testlet System GL Chooses 8 Items Site Grade Candidate Item List GL 4 & Expert 4 Items Levels Published Study List Expert Chooses 8 Items Teachers Admin Scores Captured in Testlets Numbers Testlets Created in Testlets EADMS Sent to Teachers EADMS Teachers Use Results Results Analyzed Results Harvested Formatively W/R Summative Assessment from EADMS
formative Tool • Candidate List contains 20 items that are a representative mix of the 2 or 3 Priority Standards with regard to Claims, Item Types, Task Models, & DOK levels, • Model of ideal mix built • Grade level Testlet mix compared to ideal model mix • Each Testlet awarded an integer score on a scale from 0 to 8
formative Tool Evaluation Item # 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 Q Site A X X X X X X X X 1 Site B X X X X X X X X 4 X Site C X X X X X X X 8 STND 5NF3 5NF4 5NF6 DOK DOK 1 DOK 2 DOK3 TM TM1a TM1b TM2 TM3 TM4a TM4b TM6a TM6b TM1 TM2c Claim Claim 1 Target F C2A C3D Model 2 Items 1 Item 1 or 2 Items 2 Items 1 or 2 Items
Killer Item 11 C1TF DOK-2
Killer Item 19 C2TA DOK-3
Year 2 Results Results pertain to: Extending the model presented previously Relationships among elements (describing implementation) Relationships with outcomes Caveats: Lack of true experimental control Aggregate Data Single Teslet
Year 2 Results
Year 2 Results
Recommend
More recommend
