Reporting Results from Three Dimensional Science Assessments Nathan Dadey Center for Assessment National Conference On Student Assessment San Diego, CA June 27 th , 2018
Most educational assessment score reports look and feel about the same, and have for some time. 6/28/2018 2
Look at this part of a report from 2001. 6/28/2018 3
6/28/2018 3D Science Assessment Reporting 4
Seem familiar? 6/28/2018 5
And this would be fine, if we thought score reports were working the way we want them to. 6/28/2018 6
Take Away Points 1. Consider Audience and Use – Deemphasize indefensible uses – Design early and ask people to think aloud using draft reports 2. Put psychometric concerns in context, as one of many competing concerns 3. Stop thinking in terms of two page paper or pdf score report – if possible, leverage technology 6/28/2018 3D Science Assessment Reporting 7
This presentation focuses on the first two points. 6/28/2018 8
Take Away Points 1. Consider Audience and Use – Deemphasize indefensible uses – Design early and ask people to think aloud using draft reports 2. Put psychometric concerns in context, as one of many competing concerns 3. Stop thinking in terms of two page paper or pdf score report – if possible, leverage technology 6/28/2018 3D Science Assessment Reporting 9
Part I Audience & Use 6/28/2018 3D Science Assessment Reporting 10
Score reports are infamously “the last thing developed and the first thing seen” (c.f., Zenisky, Hambleton & Sireci, 2009) 6/28/2018 3D Science Assessment Reporting 11
The Current State of Affairs Educators used the score report information for a wide range of purposes, from planning everyday lessons to making school-level programming decision (e.g., placement) or district-level decisions (e.g., offering professional learning opportunities). Although ACCESS is not intended for informing daily instruction, educators heavily relied on ACCESS scores due to lack of access to or familiarity with other resources to understand students’ English language development. 6/28/2018 12
Thus we need to show best practice around intended uses and also signal unsupported uses. 6/28/2018 13
Narrative or Audience Intended Use Story (needed Supports info) 6/28/2018 3D Science Assessment Reporting 14
Narrative or Audience Intended Use Story (needed Supports info) Narrative is about translating the claim to make sense to the intended audience. 6/28/2018 3D Science Assessment Reporting 15
Narrative or Audience Intended Use Story (needed Supports info) For parents, the narrative might not use terms like “three dimensional” or DCI, SEPs or CCCs. 6/28/2018 3D Science Assessment Reporting 16
Narrative or Supports are all Audience Intended Use Story (needed Supports info) about what needs to be put in place beyond the report to insure the outcomes based on the intended uses can be meet. 6/28/2018 3D Science Assessment Reporting 17
Narrative or Audience Intended Use Story (needed Supports info) Which gets into your theory of action that supports assessment score use, but that’s not a story for today. 6/28/2018 3D Science Assessment Reporting 18
Testing out Designs • Action plan – developing out a template based on prior table and working with states to see how it works – At several stages of development • Building on the design pattern work coming out of ETS (cf., Zapata-Rivera & Katz, 2016) 6/28/2018 3D Science Assessment Reporting 19
Part II Putting Psychometrics in its Place 6/28/2018 3D Science Assessment Reporting 20
Don’t try to solve a communication problem with a measurement model. 6/28/2018 3D Science Assessment Reporting 21
Behrens, DiCerbo, Murphy & Robinson (2013) 6/28/2018 22
Behrens, DiCerbo, Murphy & Robinson (2013) 6/28/2018 23
A Balancing Act • Psychometrics is only one concern among many, other concerns in include: – Audience – Intended Use – Constraints • Sometimes sacrificing psychometric “integrity” to better tell a story is worthwhile (e.g., using scores with no “value-added”)! 6/28/2018 3D Science Assessment Reporting 24
A Balancing Act, Continued Two perspectives on subscores: • Subscores are less reliable versions of the total score and should not be reported • Subscores provide insight into the structure of the assessment 6/28/2018 3D Science Assessment Reporting 25
There are examples of each perspective within the field. 6/28/2018 3D Science Assessment Reporting 26
Here’s an example reporting structure from the latter option. 6/28/2018 3D Science Assessment Reporting 27
6/28/2018 3D Science Assessment Reporting 28
Foregrounding • Any aspect of the standards could “foregrounded,” or emphasized, to provide additional structure to the claims, achievement level descriptors, score reports, blueprints, item clusters, or any combination thereof – For example, reporting three subscores based on the Disciplinary Core Idea domains will foreground these domains 6/25/2018 29
Foregrounding & Subscores • Should the foregrounded aspects of the assessment be reflected in: – ALDs? – Subscores? • The model used to produce the overall score and subscores doesn’t have to be the same! 6/28/2018 3D Science Assessment Reporting 30
What Defines Subscores? • A Single Dimension – Leveraging 2D coding – Probably a bad idea (even if say, a Scientific and Engineering Practice Score and Cross Cutting Concept Score are reported within a Content Domain) 6/28/2018 3D Science Assessment Reporting 31
What Defines Subscores? • A Single Dimension • Dimension Groupings • PEs 6/28/2018 3D Science Assessment Reporting 32
What Defines Subscores? Dimension Groupings – DCI Topic Domain, Topic Bundles – Groupings of the Scientific and Engineering Practices (Investigating, Evaluating and Developing Explanations & Solutions) – Groupings of Cross Cutting Concept Scores (Systems, Causality, Patterns) 6/28/2018 3D Science Assessment Reporting 33
Cross-tabular Score Report? Developing Investigating Evaluating Explanations & Solutions Systems Score Score Causality Score Patterns Score 6/28/2018 34
6/28/2018 3D Science Assessment Reporting 35
Supplemental Slides 6/28/2018 3D Science Assessment Reporting 36
Part III Moving Beyond the PDF 6/28/2018 3D Science Assessment Reporting 37
Leveraging digital formats • How can we use the affordances of digital technology to tell the right kinds of stories? – Tooltips, hyperlinks, tabs, etc. – A score report should be less of a “one page summary” and more of a flexible tool that scaffolds users in developing understanding (e.g., score reporting as more of an app than a single static page) 6/28/2018 3D Science Assessment Reporting 38
A example (not so great) parent report What are some How did my student What kinds of things things I can do to do? can my student do? help my student? 39
A example (not so great) parent report What are some How did my student What kinds of things things I can do to do? can my student do? help my student? This category could be a bad idea. 40
Reporting Scores from NGSS Assessments: Exploring Scores & Subscores Nathan Dadey Center for Assessment Reidy Interactive Lecture Series Portsmouth, NH September 28 th , 2017
Part III Moving Beyond the PDF 6/28/2018 3D Science Assessment Reporting 42
Leveraging digital formats • How can we use the affordances of digital technology to tell the right kinds of stories? – Tooltips, hyperlinks, tabs, etc. – A score report should be less of a “one page summary” and more of a flexible tool that scaffolds users in developing understanding (e.g., score reporting as more of an app than a single static page) 6/28/2018 3D Science Assessment Reporting 43
A example, perhaps not so great, parent report What are some How did my student What kinds of things things I can do to do? can my student do? help my student? 44
A example, perhaps not so great, parent report What are some How did my student What kinds of things things I can do to do? can my student do? help my student? 45
A example, perhaps not so great, parent report What are some How did my student What kinds of things things I can do to do? can my student do? help my student? This category could be a bad idea as the assessment many not support such uses , but pushes on the supports that could be provided. 46
Slides From a Previous Presentation an the 2017 Reidy Interactive Lecture Series. 6/28/2018 3D Science Assessment Reporting 47
And the score reports will need at least one score. For federally required accountability, likely an achievement level classification*. *ESSA also requires “individual student interpretive, descriptive, and diagnostic reports… that allow parents, teachers, principals, and other school leaders to understand and address the specific academic 6/28/2018 3D Science Assessment Reporting 48 needs of students” (§1111(b)(2)(B)(ix))
Recommend
More recommend
