DEVELOPING ACHIEVEMENT LEVELS FOR THE 2017 NATIONAL ASSESSMENT OF EDUCATIONAL PROGRESS GRADE 4 WRITING ASSESSMENT NCSA Presentation – June 2018 Tim O’Neil, Ed.D. ALS Project Director Pearson NGSS Design Challenges: One State’s Design 1
Background Current NAEP Writing Framework: Writing on Computer • Third NAEP Writing Framework since 1990; adopted in 2007 • Writing assessments under this framework administered in grades 8 and 12 in 2011 • Achievement levels for grades 8 and 12 set in 2012 • 2017 is first administration of this framework at grade 4 Pearson contracted with the National Assessment Governing Board to set achievement levels for the grade 4 NAEP writing assessment Setting Achievement Levels for NAEP Writing 2
ALS High Level Design Achievement level setting methodology • Body of Work (BoW) method • Appropriate for the item types and scaling • Completed in a reasonable amount of time • Governing Board has experience with the method • Supported by the measurement field Setting Achievement Levels for NAEP Writing 3
Body of Work Method • Panelists work directly with actual student responses and judge them in relation to the NAEP grade 4 writing Achievement Level Descriptions • Panelist judgements combined with known student performance (scores) allow for the derivation of cut scores for each panelist • Panelists judge each student body of work (BoW) as belonging to one of the defined achievement levels (e.g. Basic, Proficient, Advanced) • Panelists engage in multiple rounds of judgement informed by summary information and group discussion Setting Achievement Levels for NAEP Writing 4
Moodle We created a user interface within Moodle (open-source learning platform) with similar functionality of BoWTIE (software used for the 2011 grade 8 and 12 ALS). ‒ Solution was used prior to and throughout meetings for sharing of pre- meeting materials, panelist review of BoWs and entering ratings, extraction of rating data for analysis, providing panelist feedback, completing questionnaires, etc. ‒ Was used successfully for PARCC standard setting meetings as well as others (Bookmark and Modified Angoff) ‒ Has proven effective with pre-meeting training, materials sharing, form/survey completion, presentation of items/responses, capture of panelist ratings, presentation of feedback data, etc. Setting Achievement Levels for NAEP Writing 5
Setting Achievement Levels for NAEP Writing 6
ALS Process Overview Panelists: • Take a NAEP grade 4 writing test • Receive information about the rubric used to score responses from the NAEP grade 4 writing assessment • Receive training on understanding the NAEP Writing Framework • Develop a shared understanding of each achievement level description (ALD) • Receive training on and implement the ALS method known as Body of Work • Recommend exemplars for use in reporting • Respond to process evaluations throughout the study Setting Achievement Levels for NAEP Writing 7
ALS Activities Conducted within Moodle • Pre-meeting: introductory video, panelist information, materials review (ALDs, Briefing Booklet, Frameworks, NDA) • ALD practice classification activity • BoW practice • BoW judgments all rounds • BoW feedback all rounds • Exemplar BoW selection • Process evaluations throughout – from pre-meeting through wrap up. Every major task (11 overall) Setting Achievement Levels for NAEP Writing 8
Pre Meeting Setting Achievement Levels for NAEP Writing 9
Facilitator Materials Section Prompt 1.pdf Training Exemplar 1.pdf Prompt 2.pdf Training Exemplar 2.pdf Prompt 3.pdf Training Exemplar 3.pdf Video Prompt 3.pdf 223_Prompt 1.pdf 223_Prompt 2.pdf 2854_Prompt 1.pdf 2854_Prompt 2.pdf Setting Achievement Levels for NAEP Writing 10
Judgment Task Prompt 1 A25_Prompt 1 Prompt 2 A25_Prompt 2 Setting Achievement Levels for NAEP Writing 11
Complexity of Judgment Task Design Unique Forms Common Forms Panel Prompt Form 1 Form 2 Form 3 Form 4 Form 5 Form 6 Form 7 A1 A2 A3 A4 C1 C2 C3 1 Explain Explain Persuade Convey Explain Convey Convey A A5 A6 A7 A8 C4 C5 C6 2 Convey Convey Explain Persuade Convey Explain Persuade Form 8 Form 9 Form 10 Form 11 Form 5 Form 6 Form 7 B1 B2 B3 B4 C1 C2 C3 1 Explain Explain Persuade Persuade Explain Convey Convey B B5 B6 B7 B8 C4 C5 C6 2 Convey Convey Explain Convey Convey Explain Persuade Note: 7 BoWs selected per form (one common form has 8), results in each panel reviewing 50 BoWs (78 total) BoWs selected so that they are of roughly equal overall difficulty across panels Setting Achievement Levels for NAEP Writing 12
Composition of Tables NON- TEACHER TEACHER TEACHER TEACHER TEACHER TEACHER TEACHER TEACHER TEACHER TEACHER GENERAL GENERAL NON- GENERAL GENERAL NON- PUBLIC PUBLIC TEACHER PUBLIC PUBLIC TEACHER NON- TEACHER TEACHER TEACHER TEACHER TEACHER TEACHER NON- TEACHER TEACHER TEACHER TEACHER GENERAL GENERAL GENERAL GENERAL NON- GENERAL PUBLIC PUBLIC PUBLIC PUBLIC TEACHER PUBLIC 13 Setting Achievement Levels for NAEP Writing
Feedback Setting Achievement Levels for NAEP Writing 14
Reference Materials Prompt #1 Prompt #2 BoW BoW Order ID Title Response Title Response A25_Napoidjf 1 A25 Ylksu sl Ofnsl A25_Ylksu sl Ofnsl Napoidjf Umndp Umndp A12_Wldi fj 2 A12 Wldi fj Mhapvuh Radk Padfunv A12_Radk Padfunv Mhapvuh C4_Napoidjf 3 C4 Napoidjf Umndp Ylksu sl Ofnsl A22_Ylksu sl Ofnsl Umndp 4 A15 Radk Padfunv A15_Radk Padfunv Radk Padfunv A15_Radk Padfunv 5 C16 Wldi fj Mhapvuh C16_Radk Padfunv Ylksu sl Ofnsl C16_Ylksu sl Ofnsl A22_Napoidjf 6 A22 Ylksu sl Ofnsl A22_Ylksu sl Ofnsl Napoidjf Umndp Umndp A9_Napoidjf 7 A9 Radk Padfunv A9_Radk Padfunv Napoidjf Umndp Umndp A5_Napoidjf A5_Wldi fj 8 A5 Napoidjf Umndp Wldi fj Mhapvuh Umndp Mhapvuh A17_Wldi fj 9 A17 Wldi fj Mhapvuh Ylksu sl Ofnsl A17_Ylksu sl Ofnsl Mhapvuh C15_Napoidjf 10 C15 Radk Padfunv C15_Radk Padfunv Napoidjf Umndp Umndp Setting Achievement Levels for NAEP Writing 15
Real Time Review of Survey Results Setting Achievement Levels for NAEP Writing 16
Exemplar Selection Prompt 1 Video 1 C18_Prompt 1 Prompt 2 C18_Prompt 2 Setting Achievement Levels for NAEP Writing 17
Lessons Learned Pros • Computer- based platform is a practical necessity for managing today’s standard settings • Flexibility of Moodle has proven to be effective in handling many methodologies and design complexities (as demonstrated) • Efficiencies in terms of digital versus paper, distribution of materials, collection of judgments/ratings, and real time oversight cannot be overstated Cons • Not all users comfortable with computers • Reliance on multiple computers and management of multiple logins can be cumbersome • Security needs are greater (than paper based) • Some interface constraints can mean limitations to functionality Setting Achievement Levels for NAEP Writing 18
Questions? tim.oneil@pearson.com
Recommend
More recommend
