Data Visualization/Reporting and Valid Accountability System Design - PowerPoint PPT Presentation

Overview Validity vis-à-vis Accountability Systems Data Quality vis-à-vis Data Use Data is not equal to Information The Medium is the Message References Data Visualization/Reporting and Valid Accountability System Design Damian W. Betebenner National Center for the Improvement of Educational Assessment Dover, NH CCSSO National Conference on Student Assessment Orlando, Florida June 21st, 2011 Damian W. Betebenner Data Visualization/Reporting and Valid Accountability System Design

Overview Validity vis-à-vis Accountability Systems Data Quality vis-à-vis Data Use Data is not equal to Information The Medium is the Message References Topics of Today’s Presentation Validity vis-à-vis Accountability Systems. Data Quality vis-à-vis Data Use. Data � Information . The Medium is the Message. Damian W. Betebenner Data Visualization/Reporting and Valid Accountability System Design

Overview Validity vis-à-vis Accountability Systems Data Quality vis-à-vis Data Use Data is not equal to Information The Medium is the Message References Validity & Accountability (Braun 2008) Validity as a scientific enterprise Validity is an ongoing argument that seeks to clarify what a measurement means and to understand the limitations of each score interpretation. (Cronbach,1988) Validity as consideration of consequences Validity is an overall evaluative judgment, founded on empirical evidence and theoretical rationales, of the adequacy and appropriateness of inferences and actions based on test scores. (Messick, 1989) Damian W. Betebenner Data Visualization/Reporting and Valid Accountability System Design

Overview Validity vis-à-vis Accountability Systems Data Quality vis-à-vis Data Use Data is not equal to Information The Medium is the Message References Systemic Validity (Braun, 2008) “Assessment practices and systems of accountability are systemically valid if they generate useful information and constructive responses that support one or more policy goals (Access, Quality, Equity, Efficiency) within an education system, without causing undue deterioration with respect to other goals.” “In validating an accountability system, the theory-of-action plays the same role as does the construct in test validation.” “Consequential validity is the ultimate criterion by which we should judge an accountability system.” Damian W. Betebenner Data Visualization/Reporting and Valid Accountability System Design

Overview Validity vis-à-vis Accountability Systems Data Quality vis-à-vis Data Use Data is not equal to Information The Medium is the Message References Theory of Action What is the theory of action that turns data into information that increases educational efficacy? Articulating that theory of action is essential to validating the teacher evaluation system. Example 1: Publishing VAM scores in the LA Times. Example 2: Incentivizing behavior through monetary compensation (see http://www.youtube.com/watch?v=u6XAPnuFjJc ). Perhaps this is what Box was referring to when he said, “All models are wrong, some are useful.” Damian W. Betebenner Data Visualization/Reporting and Valid Accountability System Design

Overview Validity vis-à-vis Accountability Systems Data Quality vis-à-vis Data Use Data is not equal to Information The Medium is the Message References Creating a Virtuous Cycle Assessment Data Quality Assessment Data Use Damian W. Betebenner Data Visualization/Reporting and Valid Accountability System Design

Overview Validity vis-à-vis Accountability Systems Data Quality vis-à-vis Data Use Data is not equal to Information The Medium is the Message References The Data to Knowledge Continuum Damian W. Betebenner Data Visualization/Reporting and Valid Accountability System Design

Overview Validity vis-à-vis Accountability Systems Data Quality vis-à-vis Data Use Data is not equal to Information The Medium is the Message References From Paper to Interactive Media We are witnessing a revolution in the way in which humans interact with media. From static (paper) to interactive (Web 2.0) Confirmatory versus Exploratory data analysis. Causal Attribution (man as researcher, intra-psychic) versus Causal Explanation (conversational, inter-psychic). Rendering verdicts about education/educator quality or promoting conversations about education/educator quality. UX (User Experience) Design: What use cases do we want to promote? (Telling versus Involving). Tell me and I will forget, show me and I will remember, involve me and I will understand. (Confucius) Damian W. Betebenner Data Visualization/Reporting and Valid Accountability System Design

Overview Validity vis-à-vis Accountability Systems Data Quality vis-à-vis Data Use Data is not equal to Information The Medium is the Message References References Braun, H. I. (2008). Viccissitudes of the validators. Presentation made at the 2008 Reidy Interactive Lecture Series, Portsmouth, NH, September, 2008. Damian W. Betebenner Data Visualization/Reporting and Valid Accountability System Design