http://www.springer.com/us/book/9783319673035 The Springer Series in - PowerPoint PPT Presentation

http://www.springer.com/us/book/9783319673035

The Springer Series in Measurement Science and Technology The Springer Series in Measurement Science and Technology comprehensively covers the science and technology of measurement, addressing all aspects of the subject from the fundamental principles through to the state-of-the-art in applied and industrial metrology, as well as in the social sciences . Volumes published in the series cover theoretical developments, experimental techniques and measurement best practice, devices and technology, data analysis, uncertainty, and standards, with application to physics, chemistry, materials science, engineering and the life and social sciences.

Discontinuous Levels of Complexity in Coherent Educational Measurement: The Roles of KidMaps, Wright Maps, and Construct Maps William P. Fisher, Jr. University of California, Berkeley, CA, USA BEAR Seminar UC Berkeley Graduate School of Education 30 January 2018

Thanks to colleagues • Emily Oon and Mei Zhou at University of Macau • Fisher, W. P., Jr., Oon, E. P.-T., & Zhou, M. (2018). Assessment coherence across information complexity contexts: Coordinating classroom and international assessments. Journal of Educational Measurement, in review. • Mark Wilson at University of California, Berkeley • National Research Council. (2006). Systems for state science assessment (M. R. Wilson & M. W. Bertenthal, Eds.). Washington, DC: The National Academies Press. • Wilson, M. (Ed.). (2004). National Society for the Study of Education Yearbooks. Vol. 103, Part II: Towards coherence between classroom assessment and accountability. Chicago, Illinois: University of Chicago Press.

The problem of coherence in educational assessment • Wilson (2004; NRC, 2006) asks • What kind of information infrastructure is needed to coherently coordinate meaningful and comparable formative, interim, and summative assessments within and across classrooms? • Applications and reports would have to function within common frames of reference across developmental, horizontal, and vertical comparisons.

Developmental, horizontal and vertical forms of coherence

Coherence: Forced conformity, or an unexplored alternative? • Moss (2004), in a chapter included in Wilson’s (2004) NSSE Yearbook , fears that coherence in educational assessment will become another instance of a “high modern” scheme that systematically homogenizes human variation into bureaucratically manageable forms. • She cites Scott’s (1998) account of the history of failed governmental efforts at improving the human condition, but does not mention Scott’s concluding suggestion that language could provide a model for a new kind of standard that functions as a means of continually adapting broad principles to novel circumstances.

Multiple levels of complexity in language and information infrastructures • Language is, Scott (1998, p. 357) says, “a structure of meaning and continuity that is never still and ever open to the improvisations of its speakers ." • Star and Ruhleder (1996) similarly point out that "The competing requirements of openness and malleability, coupled with structure and navigability, create a fascinating design challenge—even a new science ." • The design of information infrastructures providing both structure and openness "is highly challenging technically , requiring new forms of computability that are both socially situated and abstract enough to travel across time and space “ (Star & Ruhleder, 1996, p. 132).

Levels of complexity in language (Star & Ruhleder, 1996; following Bateson, 1972) • Denotative: factual and local • The cat is on the mat. • You answered 28 of the 50 questions correctly. • Metalinguistic: abstractly refers to words • The word ‘cat’ has no fur. • Your score of 28 means you fail the course. • Metacommunicative: statements about statements • My telling you where to find your cat was friendly. • My giving you a failing grade based on your score of 28 is justified given the content and difficulties of the questions.

Levels of complexity in language • “The cat on the mat” points at something real and tangible. • Pointing at the word ‘cat’ refers to an abstract concept. • It applies to all small domesticated felines. • It has an invariant meaning in the English language. • It came into use via an evolutionary process not controlled by any person or group.

Levels of complexity in language • The score of 28 on the assessment points at something real and tangible: questions answered correctly and incorrectly. • The number word ‘28’ is supposed to be abstract. • But the meaning of an assessment score of ‘28’ is tied to a particular set of questions. • It means something different across tests. • It came into use via a process controlled by an individual person or group. • Used to indicate a learning outcome, ‘28’ does not have a general and invariant meaning. • The theoretical justification for failure based on the score is contained in a privately organized information system not open to contestation or confirmation by others.

What happens when we ignore levels of complexity in language? • We find ourselves: • “…with organizations which are split and confused, systems which are unused or circumvented, and a set of circumstances of our own creation which more deeply impress disparities on the organizational landscape" (Star & Ruhleder, 1996, p. 118). • Sounds like Scott’s (1998) history of failed “high modern” schemes • Also resonates with Ladd’s (2017) documentation of the flawed U.S. NCLB proficiency standards. • Ladd, H. F. (2017). No Child Left Behind: A deeply flawed federal policy. Journal of Policy Analysis and Management, 36 (2), 461-469.

What’s the alternative? • Can number words be connected with concrete observations and abstract meanings that remain invariant throughout the language? • Can number words emerge from a group-level process not under the control of any individual? • Can publicly reproducible justifications for uses of number words provide independent validation of the inferences made?

Levels of complexity in education • Denotative: statements about learning • You answered these questions correctly and incorrectly. • Your score on the test was a particular count of correct responses. • Kidmap display • Metalinguistic: learning about learning • We observe a pattern of consistently increasing difficulty in items. • Similar patterns of invariance emerge across assessments. • Wright map display • Metacommunicative: theories about learning • We see item features that cause items to be easy or hard. • We design tests from specifications, and they function as expected. • Construct map display and construct specification equation

Levels of complexity in education • Denotative: Concrete statements about learning • You answered these questions correctly and incorrectly. • Your score on the test was 28. • Kidmaps HARD EASY CORRECT INCORRECT

Levels of complexity in education • Metalinguistic: Abstract learning about learning MEASURE | MEASURE <more> --------------------- PERSON -+- ITEM ----------------- <rare> 7 .## + 7 | • We observe a pattern of self-organized | | 6 + . 6 | # conjoint order: | | . 5 . + 5 | # . | • consistently increasing item difficulties, and .## | . 4 . + ### 4 . | ## • consistently increasing student abilities. .### |T ####. . | #### 3 .##### T+ # 3 • Similar patterns of spontaneous .## | ###### .####### | ###### .####### | ######## 2 .######## + ###### 2 invariance emerge across tests. .########## |S ################. .#################### S| ##################. .############# | ################ 1 .################# + ###################### 1 .############## | ##################### • Wright maps #################### | ################# .######################## | #######################. 0 .######################## M+M ###########################. 0 .########################### | ###################### • Equating .######################## | ####################### .####################### | ############### -1 .######################### + ######################. -1 .##################### | #################### • Item banks .############## S| ##############. .############# |S #############. -2 .########### + ###############. -2 .###### | ####. .###### | ########## .#### | ########. -3 .### T+ #####. -3 .#### | #### .# |T ## . | #. -4 . + . -4 | # . | . . | -5 + -5 . | | | -6 . + -6 <less> --------------------- PERSON -+- ITEM ----------------- <freq>