Establishing Performance Standards for School Accountability Systems Chris Domaleski Center for Assessment National Conference on Student Assessment San Diego, CA June 27, 2018
Accountability Ratings: Meaning Matters • How can we help stakeholders understand, interpret, and make good decisions based on school ratings? • Whether these ratings are points, stars, letters, or something else, establishing and communicating the ratings thoughtfully is an important part of accountability. Performance Standards in Accountability 2 CCSSO NCSA 2018
Everyone knows what an ‘A’ means, right? • Credit Ratings – S&P describes an “A” as, “strong capacity to meet financial commitments” a s distinguished from AAA “extremely strong” or AA “very strong” – In August 2011 S&P downgraded the U.S. from AAA to AA+ stating in part, “the downgrade reflects our view that the effectiveness, stability, and predictability of American policymaking and political institutions have weakened.” – Does the letter grade alone tell you what you need to know to inform a buy or sell decision? • New York Department of Health restaurant grades – “A letter grade of A indicates the a restaurant received 13 or fewer health code violations in their last inspection … violations are likely to be technicalities or non- critical infractions” – Does the letter grade alone provide the information you need to make a decision about where to dine? Performance Standards in Accountability 3 CCSSO NCSA 2018
The meaning of a rating matters • Other interesting ratings: – Rotten Tomatoes Movie ratings: • Fresh = Tomatometer of at least 60% positive reviews • Certified Fresh = 75% positive reviews. – Hurricanes: • Saffir-Simpson Hurricane Scale rates a hurricane's intensity using wind speed and storm surge. • Category 5 is characterized by 155+ wind speeds and storm surges above 18 feet – Collectables: • Typical scale: mint, near mint, excellent, very good, good, poor. • For example, a “very good” baseball card can have scratches, stains, creases, and fading. • The point is: meanings matter! – Should you spend your money on a movie that isn’t ‘Certified Fresh’? – Should you go outside in a category 3 hurricane? – Can you expect a baseball card in ‘good’ condition to have value? Performance Standards in Accountability 4 CCSSO NCSA 2018
Performance Standards in Accountability • Given the importance of accountability ratings, it is reasonable to require evidence that the rating has a high degree of validity for the intended uses and purposes • A substantial part of that validity argument is the design and implementation of sound process for establishing standards that credibly reflects the state’s vision and policy priorities • While there is a research base for assessment standard setting, accountability standards are often overlooked and are defined arbitrarily or tautologically • Ultimately, the intent is to improve the clarity and usability for stakeholders Performance Standards in Accountability 5 CCSSO NCSA 2018
Principles • Rationale, procedure, and results should be well documented • The process should be designed to allow well- qualified policy-makers, experts, and/or stakeholders an opportunity to inform the decisions • Participants should be provided information associated with relevant consequences and criteria Performance Standards in Accountability 6 CCSSO NCSA 2018
How to define ‘Good Enough’ Performance? • Decisions of policy-makers and leaders, which are often reflected in law, rules, guidance, or vision statement – SEA long-term goals and measures of interim progress – ESSA high school graduation rate criterion of 67% • Based on normative data – E.g. top 10% receive an ‘A’ – Analyses of legacy data to determine rate of improvement that can be expected • Expert judgment • Relationship to external criteria Performance Standards in Accountability 7 CCSSO NCSA 2018
Promising Practices • Develop policy descriptors for each level • Develop school performance level descriptors • Convene panel or other group (e.g. task force, advisory group) to: – Evaluate school profiles with respect to descriptors – Review data on achievement patterns – Set Indicator thresholds and overall thresholds • Evaluate • Document • Produce material to support interpretation and use Performance Standards in Accountability 8 CCSSO NCSA 2018
State Illustrations • Today we will hear from two states that implemented a systematic process to set meaningful performance expectations for their school accountability system • The intent is to learn from their experiences to identify challenges and opportunities to inform the work in your state Performance Standards in Accountability 9 CCSSO NCSA 2018
Recommend
More recommend
