BEYOND ITEM REVIEW AND RANGEFINDING: INNOVATIVE WAYS STATES ENGAGE - PowerPoint PPT Presentation

BEYOND ITEM REVIEW AND RANGEFINDING: INNOVATIVE WAYS STATES ENGAGE EDUCATORS TO DEVELOP AND IMPLEMENT ASSESSMENTS Charity Flores, Ed.D. Indiana Department of Education Jan Reyes, Ed.D Georgia Department of Education Lisa Sireno Missouri Department of Elementary & Secondary Education Holly Baker, PhD Data Recognition Corporation Corey Palermo, PhD Measurement Incorporated

Introduction • Welcome • Background • Presenters • Feedback • socrative.com OR • download socrative student app • Select login – student login • Room name = PALERMO805

Lisa Sireno Standards and Assessment Administrator Engaging Educators in Assessment Development June 2019

Why engage Missouri educators?

Show-Me Success

Assessment Implementation Schedule 2016-17 2017-18 2018-19 2019-20 22 ELA & Math English Language Arts & Mathematics Science Field Test Science Science Social Studies Social Studies Social Studies Field Test Personal Finance Personal Finance Assessments Aligned to the Previous Expectations Assessments Aligned to the New Expectations

What did they do?

Building Missouri Assessments

Educators and Missouri Assessments Educators representing over 250 Missouri school districts and charter LEAs participated in the development of Missouri’s newest statewide tests.

Recruiting • Content-area collaboratives and work groups • Listservs • Nominations from area supervisors • Nominations from Missouri educator and administrator organizations

Challenges Benefits • Mix of participants • Access to classroom experiences • Competing activities • Diverse perspectives • Logistics • Educator ownership • Paperwork! • Professional learning • Standards implementation • Peer review evidence

What did we learn? • Establish the purpose • Limit jargon • Be flexible • Expect the unexpected • Some tasks are more difficult, but worth the effort…

Questions? Lisa.Sireno@dese.mo.gov 573-751-3545

IN: Why engage educators? • Educator contributions to assessment development have a historical practice in Indiana • Blueprint Development • Content and Fairness Review • Standard Setting • Recent legislation and assessment changes shifted increased engagement • Legislation: scoring of open ended items • Requires the use of IN educators to the extent possible • Contributions to promote stakeholder involvement • Specifications • Data Review • Range Finding and Rubric Validation

IN: Process for involving educators • Recruitment of educators varies on the process • Nominations from superintendents • Standard setting • Volunteers with independent school or district confirmation • Content development processes • Volunteers with third-party screening, training and oversight • Scoring

IN: Challenges • Communication • Difficult for IN to communicate directly with educator population • Timing • Many efforts overlap with challenging times in school calendars or summer breaks • Establishing protocols • Reliance on criteria and revisiting this after each cycle to continue to refine the criteria

IN: Benefits • Professional Development • Internalizing quality processes • Development of assessment • Scoring of items based on a rubric • Assessment literacy • Alignment to instructional practices • Educators are content experts • Highlight dimensions of standards as they are used in daily instruction

IN: Lessons learned • Recruitment • Be proactive in recruiting — it always takes longer than you think! • Communicate with local organizations • Who participated, what they did, how they can apply this to their daily work • Be intentional in procurement efforts • Clearly define where it is expected for educators to be involved so that the schedule can support these efforts

GA: Why engage educators? • Georgia has a longstanding history of involving educators in various aspects of assessment development. • Educator involvement is not a legislative mandate, but is considered best practice. • Assessment Peer Review guidance indicates that test items should be: • reviewed by an external committee for alignment to academic content standards, intended levels of cognitive complexity, intended levels of difficulty, construct-irrelevant variance, and consistency with item specifications; and • reviewed with field test data to evaluate the quality of items and select items for operational use.

GA: Process for involving educators • Nominations are solicited from districts each year. • System Test Coordinators, Curriculum Coordinators, Special Education Directors • All nominees are asked to submit an application with additional background information. • Invitees are selected to provide a balanced representation based on: • Gender • Ethnicity • Region • Experience • Special populations • Educators are involved in the development of all state assessments, including formative assessments for our youngest learners.

GA: Process for involving educators Georgia Keenville Georgia Georgia Alternate Kindergarten (Game-Based Milestones Assessment Inventory of Assessment for Developing Skills Grades 1 and 2) Blueprint Assessment Development Development Development Specifications Committee Committee ALD Development Blueprint Task Review Playtesting Development Extended Dashboard New Item Review Piloting Standards Review Review Item Matching Rangefinding ALD Development Calibration Study Workshops Inter-rater Data Review New Item Review reliability study Standard Setting Data Review Standard Setting

GA: Challenges • Communication • District communication to teachers • Recruitment • Fewer nominations for some programs • Declines/cancellations • Balanced representation • Timing • Available meeting space • Conflicting obligations for teachers and other Department staff • Compressed field test scoring window for constructed-response • Differences of opinion from one committee to the next

GA: Benefits • Buy-in from educators • “I felt we were contributing to a process that allows teachers to feel assured that these assessments are valid and fair to all of our students.” • “It was a great comfort to me to know that so many steps are taken and that educators have a role in that process.” • “The people who create our tests really do care about the children and try to create the most fair and reliable questions for all students.” • Professional development for educators • “The process was incredible. It's probably one of the most beneficial experiences of my teaching career.” • “It was interesting to collaborate with others from around the state. We don't often get that opportunity.” • Ensure alignment and expectations • “It was refreshing to learn how much attention was afforded to ensuring that the questions met the expectations of student learning in accordance to the GSE.” • “This was a great way to ensure that the test items are valid.”

GA: Lessons learned • Frequent communication • Districts • Educators • Be more transparent • All teachers would benefit from a better understanding of the assessment development process

Discussion ● Requirements for participation How does this influence the activities in which teachers are ○ engaged? How has the state selected teacher participation in some but not other aspects of the program? What other opportunities exist within your programs where there ○ could be value in involving teachers? ● Participation as professional development ○ Instructional impact - how do we measure this, is it important? ○ If valuable, how can this be broadly disseminated to encouarge participation? ● Common challenges - how do we overcome them?