Windows Into The Mind Richard J. Shavelson & Maria Araceli Ruiz-Primo Stanford University Invited Talk University of Ancona Ancona, Italy July ????, 2000
Assessing Knowledge Structure DENNIS THE MENACE “GEE! I’M GLAD WE DID CONCEPT MAPS - NOW WE BOTH KNOW WHAT I LEARNED TODAY!!! ” July ????, 2000 Windows Into The Mind 2
Overview • Cognitive theory underlying assessments of knowledge structure • Direct and indirect methods of assessing “cognitive structure” • Direct method: Concept maps • Indirect method: Cognitive maps • Link between concept and cognitive maps • Concluding comments July ????, 2000 Windows Into The Mind 3
Toward An Achievement Framework: Knowledge Types and Dimensions Declarative Procedural Strategic Knowledge Knowledge Knowledge (Knowing the “ that ”) (Knowing the “ how ”) (Knowing the “ which ,” Proficiency “ when ,” and “ why ”) Low High Extent ( How much? ) Domain-specific content: Domain-specific Problem schemata/ Structure • facts production strategies/ ( How is it organized? ) • concepts systems operation systems • principles Others (Precision? Efficiency? Automaticity?) Cognitive Cognitive Tools: Tools: Planning Planning Monitoring Monitoring July ????, 2000 Windows Into The Mind 4
Associative and Semantic Models of Memory • Nodes represent concepts Is a • Lines represent Contains relations among concepts Is an example of • Probe memory by asking about concepts (or lines) Associative Semantic July ????, 2000 Windows Into The Mind 5
What Are Concept Maps? A Concept Map of a Concept Map! Concept Map Map Concept is a graph consist of consist of labeled nodes linked by lines denote a represent between a pair of concepts relation July ????, 2000 Windows Into The Mind 6
Concept Map Assessment: An Example Ayita, I think the student had to simply write in a number Task Task (1-5) on the line from words given The following map shows the relationship among concepts related to genetic continuity. The concepts are arranged hierarchically and linked each other. Please examine the map and supply, in the space provided, a word or two for labeling each link such that association between concepts is made clear. Response Format Scoring System Response Format Scoring System Genetic • One point is given for each correct relationship continuity • One point is given for each level up until two levels beyond the last branching Asexual Sexual reproduction reproduction • One point is given for the first branching where two or more concepts are connected to the concept above. Three points are given for any subsequent branching Meiosis 1. through • One point is given for each cross link 2. producer 3. requires 4. called Female Male 5. fuse into gamete gamete Source: Tamir, P. (1995). Science Assessment. In M. Birenbaum & F. J. R. C. Dochy (Ed.). Alternatives in assessment of achievements, learning processes, and prior knowledge . Boston: Kluwer Academic Publishers. July ????, 2000 Windows Into The Mind 7
Are There Different Types of Concept Maps? Yes, more than we would like! Yes, more than we would like! Variations in the task, the response format, and the scoring system produce different concept mapping techniques. July ????, 2000 Windows Into The Mind 8
Construct-A-Map Construct a concept map that reflects what you know about what a concept map is. Organize the terms in relation to one another in any way you want. understanding demands relationships knowledge clear explanation of determine the acquisition demands structure of a understanding the meaning of concepts stand for can be represented to some extent by a nodes with explanations stand for with explanations show the essential relationship between composed of composed of proposition which elementary concept unit is map composed of linking lines July ????, 2000 Windows Into The Mind 9
Scoring System Based on a criterion map, it focuses on: • Proposition validity: Accuracy of propositions • Proposition validity: • Congruence : Proportion of valid propositions in student’s • Congruence map to total possible valid propositions in criterion map • Salience: Proportion of valid propositions to total number • Salience: of valid propositions in student’s map • Nodes: Concepts used in the student’s map • Nodes: July ????, 2000 Windows Into The Mind 10
Proposition “Quality” Quality of Definition Proposition 4 points Excellent Excellent Outstanding proposition. Complete and correct. It shows a deep understanding of the relation between the two concepts. acids-compounds: < that gives off H+ when dissolved in water are Good Good Complete and correct proposition. It shows a good understanding 3 points of the relation between the two concepts. acids-compounds: > are examples of Incomplete but correct proposition. It shows partial understanding 2 points Poor Poor of the relation between the two concepts. acids-compounds: < form Don’t Care Don’t Care 1 points Although valid, the proposition does not show understanding between the two concepts. acids-compounds: > is a different concept 0 points Incorrect proposition. Invalid Invalid acids-compounds: > made of July ????, 2000 Windows Into The Mind 11
Fill-In-The-Map Examine the map, the blank nodes and the terms provided on the list below. Select the term that corresponds to each node and write it down inside the circle. Use each term only once. concept map List of Concepts: • concepts is a • graph • labeled lines • propositions Score right or wrong with with with that describe the relationship nodes linked by between two nodes are called represent July ????, 2000 Windows Into The Mind 12
Fill-In-The-Linking-Lines Technique ELEMENTS, ATOMS, AND COMPOUNDS C Elements are organized on a Periodic Table are composed of the same type of Atoms are located on the left are located on the side of the upper right side are combined with equal number of electrons of the chemically and protons have a neutral are composed to form are elements of more than located in the are negatively one type of charged center of the particles of Charge have a negative have a Compounds have either a positive Transition Electrons positive or a have a Metals negative negative are formed when atoms lose or gain Ions Metals containing Non-Metals are composed of two made up of molecules and sharing opposite charged electrons are composed of are called metals and will form Ionic Molecular Compounds Compounds tend to form Polyatomic Cations Anions Ions have to have are composed of hydrogen contain one H+ and or more N 2 O 4 is a ternary ionic with two Acids compound that elements are contains sulfite Ternary Ionic which is a Compounds Binary Ionic is an example of a July ????, 2000 Windows Into The Mind 13 Compounds Sodium Sulfite
On The Reliability & Validity of Concept Maps • Reliability: Can students’ concept maps be consistently scored by different persons? • Equivalence of Mapping Techniques: Do all mapping techniques provide the same picture about students’ declarative knowledge structure? • Task Demands: Do different mapping techniques impose different cognitive demands to students? • Comparing Concept Maps With Traditional Assessments: Do concept maps measure something different than multiple-choice test? July ????, 2000 Windows Into The Mind 14
Tasks Mapping Technique 2 Mapping Technique 1 No-Concepts Provided Concepts Provided • Construct a concept map about Ions, • Construct a concept map about Ions, Molecules, and Compounds. Molecules, and Compounds. • Select another 7 concepts that you • Examine the 10 concepts and construct think are the most important in the map. explaining the topic. • Organize the concepts • Organize the concepts • Redraw the map so someone else can read it. July ????, 2000 Windows Into The Mind 15
Generalizability(Reliability) of Scores Percent of Total Variability and G Coefficients for Proposition Validity Total Scores No-Concepts, Sample A & B Sample A, and B 71.64 78.67 Person (p) .15 0 Rater (r) 0 0 Sample (s) 0 .79 p x r 22.81 17.64 p x s .01 .18 r x c 5.37 2.69 prc,e .89 .88 ρ 2 (n r =2,n s =3) (n r =2,n s =2) .89 .88 Φ 2 ρ .78 (n r =1,n s =1) .78 Φ July ????, 2000 Windows Into The Mind 16
Discriminant Validity Correlation Between Multiple-Choice Test Score and Concept Map Scores Type of No-Concepts Sample A Sample B Score Proposition Validity .64 .63 .58 Congruence .66 .55 a Salience .61 .50 .45 July ????, 2000 Windows Into The Mind 17
Recommend
More recommend
