STATE ASSESSMENTS: HOW THEY ARE CREATED OFFICE OF EDUCATIONAL - PowerPoint PPT Presentation

STATE ASSESSMENTS: HOW THEY ARE CREATED

OFFICE OF EDUCATIONAL ASSESSMENT AND ACCOUNTABILITY (OEAA) PRESENTERS Linda Howley (Test Development Manager) Scott Koenig (Social Studies Assessment Consultant) Julie Murphy (K-5 Assessment Specialist) Kyle Ward (Mathematics Assessment Coordinator)

• Improved understanding about how M-STEP assessments are developed • Gain assessment ideas and structures to take back to your TAKEAWAYS districts to help you develop your local assessments • Opportunity and/or share information to apply and be a part of our future development committees

THE LIFE CYCLE OF A M-STEP ITEM

THIS IS THE MICHIGAN ITEM BANK SYSTEM (IBS)

THE LIFE CYCLE

ITEM WRITER Standard Assignment Item Idea and Options Content Lead Feedback

ITEM VIEW – CONTENT REVIEW CAC Comments

ITEM VIEW – BIAS REVIEW BSC Comments

ITEM DATA REVIEW DATA Committee Comments Item Committee Comments with On Statistics Statistics

ITEM BANK SYSTEM – ACCEPTED ITEM

ITEM BANK SYSTEM – ACCEPTED ITEM

ITEM WRITING BASICS

ITEMS SHOULD • assess a content standard and/or benchmark. • Include accurate content information • be grade level appropriate including: appropriate cognitive level, reading levels, and thinking skills • be equitable based on gender, ethnicity, race, social class, disability, geography, etc. • be fair, clear, concise, and free of cueing • avoid emotionally charged topics (natural disasters, illness, divorce, loss of jobs) unless standard and/or benchmark specifies it • adhere to the MDE approved style guidelines • contain graphics that are clear, relevant, accurate and necessary • be reviewed and approved by content review committees and bias/sensitivity committees before field testing • require a professional editor to review items for verification before field testing

• propose a single problem, although the solution may require more than one step • avoid repeated words from the stem in the answer choices MULTIPLE CHOICE (MC) AND TECHNOLOGY ENHANCED (TE) ITEMS • restrict negatives “not” or “none” SHOULD • have distractors that are in a logical order, avoid clues, are comparable in length, complexity, and grammatical form • make certain a distractor is not a possible answer

MULTIPLE CHOICE (MC) AND TECHNOLOGY ENHANCED (TE) ITEMS

• clearly state how student is to respond • allow for a variety of acceptable answers • allow for enough time and space to respond CONSTRUCTED RESPONSE (CR) ITEMS SHOULD • conform to item development specifications and be scaffolded if necessary • be measured according to a scoring rubric that is defensible

CONSTRUCTED RESPONSE (CR) ITEM

CONSTRUCTED RESPONSE (CR) ITEM Carefully read the passage. Then complete the task that follows. My Trip to the City Aquarium Most of the class was excited to find out we were going on a field trip to an aquarium. I was not. I had never been to an aquarium, so I didn’t know what to expect. But something about the idea of spending the day looking at giant tanks filled with slimy creatures and fish did not excite me. I knew I had to go; plus, my parents had always told me to learn about new things. I agreed with them. I decided I would try to be positive. I liked the idea of learning something new. We boarded the bus and headed downtown, where the aquarium was located. When the bus pulled up to the building, I thought we were just stopping for a minute. Then I saw the sign that read “City Aquarium.” I was amazed at the size of the building. It was massive with giant white pillars and wide stone steps. I wondered why they put fish in such a big building. It looked more like a state capitol building than an aquarium. . . . . . The passage tells about a student taking a trip to an aquarium. Write an essay analyzing how the student changes his or her opinion about the aquarium trip. Use evidence from the passage to support your essay.

CONSTRUCTED RESPONSE (CR) ITEM

RESOURCES: RUBRIC FOR CONSTRUCTED RESPONSE (CR) ITEM

RESOURCES: EXAMPLES OF TEXT DEPENDENT ANALYSIS (TDA) ITEMS You can find examples of constructed responses on the M-STEP Website under Content Specific Information http://www.michigan.gov/mde/0,4615,7-140-22709_70117-350540--,00.html

COMMITTEE REVIEWS

THE COMMITTEE REVIEW GAME: POOR EXAMPLE VS. REVISED EXAMPLE EXAMPLE A EXAMPLE B

THE COMMITTEE REVIEW GAME: POOR EXAMPLE VS. REVISED EXAMPLE POOR EXAMPLE REVISED EXAMPLE

THE COMMITTEE REVIEW GAME: POOR EXAMPLE VS. REVISED EXAMPLE EXAMPLE B EXAMPLE A

THE COMMITTEE REVIEW GAME: POOR EXAMPLE VS. REVISED EXAMPLE REVISED EXAMPLE POOR EXAMPLE

THE COMMITTEE REVIEW GAME: POOR EXAMPLE VS. REVISED EXAMPLE EXAMPLE A EXAMPLE B

THE COMMITTEE REVIEW GAME: POOR EXAMPLE VS. REVISED EXAMPLE POOR EXAMPLE REVISED EXAMPLE

THE COMMITTEE REVIEW GAME: POOR EXAMPLE VS. REVISED EXAMPLE EXAMPLE A EXAMPLE B

THE COMMITTEE REVIEW GAME: POOR EXAMPLE VS. REVISED EXAMPLE POOR EXAMPLE REVISED EXAMPLE

THE COMMITTEE REVIEW GAME: MATCH THE STANDARD 6.SP .A. Develop understanding of statistical variability. 6.RP .B.3 Use ratio and reasoning to solve real-world and mathematical problems.

THE COMMITTEE REVIEW GAME: MATCH THE STANDARD Answer: 6.RP.B.3 Use ratio and reasoning to solve real-world and mathematical problems.

THE COMMITTEE REVIEW GAME: MATCH THE STANDARD 5U3.1.06 Identify the role that key individuals played in leading the colonists to revolution, including George Washington, Thomas Jefferson, Benjamin Franklin, Patrick Henry, Samuel Adams, John Adams, and Thomas Paine. 5U3.1.02 Describe the causes and effects of events such as the Stamp Act, Boston Tea Party, the Intolerable Acts, and the Boston Massacre. 4G2.0.01 Describe ways in which the United States can be divided into different regions (e.g., political regions, economic regions, landform regions, vegetation regions).

THE COMMITTEE REVIEW GAME: MATCH THE STANDARD Answer: 5U3.1.06 Identify the role that key individuals played in leading the colonists to revolution, including George Washington, Thomas Jefferson, Benjamin Franklin, Patrick Henry, Samuel Adams, John Adams, and Thomas Paine.

BLUEPRINTS, TEST MAPS, AND RENDERING

TEST DEVELOPMENT • Guided by a set of test specifications • Universal design describes the use of test formats that allow tests to be taken without adaptation by as broad a range of individuals as possible • Summative forms in Michigan • Online fixed form (K-2 Benchmarks in ELA and Math, MI-Access) • Online Computer Adaptive (M-STEP) • Paper/pencil (M-STEP, MI-Access) • Accommodated forms (M-STEP, MI-Access)

TEST DEVELOPMENT • All items align to Michigan’s content standards • Item selection follows Psychometric guidelines • p-values between .2 and .9 • item total correlations > .2 • Items selected must cover a wide variety of questions • Avoid items rejected by CAC, BSC (should be marked DNU) • Items must not be previously been released

TEST BLUEPRINTS • Original test blueprint approved by MDEs Test Development and Psychometric unit • Use original, approved blueprint for valid score reporting and interpretation • Content Lead creates modified blueprint each testing cycle • Ensure variety of standards assessed per domain while maintaining points per domain consistency year-to-year • Pre-selected anchor (equating) items from prior year • Blueprint dictates # of operational items and # of field test items • Length of test vs testing time – big consideration • Content Lead selects item type (CR, MC, TE) and mode (Online/P-P)

BLUEPRINTS AND FORMS – THE BEGINNING

BLUEPRINT DETAILS

BLUEPRINT LAYOUT

IBS MAGIC • Once blueprint is completed, Content Lead selects ‘IBS generate’ to build the test forms containing specific items • IBS designed to select items following pre-determined formula • Content Leads review each item on each form to: • Prevent clueing • Replace an item that is too similar to another on the form • Check that answer keys are comparable on each form • Content Lead has discretion to replace any and all items • Upon Content Lead approval, test map submitted for Psychometric review and approval

IBS GENERATES TEST MAP

A TEST FORM

TEST ITEM/FORM REVIEW • After Psychometricians approve test map, it is exported to DRC • DRC renders each item according to MDE-approved style guide, in the online test delivery system and delivers to OEAA • OEAA Rendering Lead and Content Lead review each item in the way they will appear to students, comparing each rendered item to the item in IBS • If changes are needed, the item goes back to DRC for re-rendering • Once all items in a program, grade, and content are approved, DRC creates forms in the delivery engine • Content Lead reviews forms for accuracy, using the IBS test map • Examples of rendered items are in Sample Item Sets

THE IMPORTANCE OF MICHIGAN EDUCATORS

Assessments developed by Michigan educators WHY we need local experts? for Michigan classrooms. - Local Leaders are our Experts - Locals assist “buy in” to the importance of assessment - Sound practice for assessment development - Fair equitable assessments