Measuring Content Knowledge of Middle Grades Teachers AAPT Summer - PowerPoint PPT Presentation

Measuring Content Knowledge of Middle Grades Teachers AAPT Summer Meeting 2009 Ann Arbor, MI Sean Smith Horizon Research, Inc.

Session Overview • The assessment in context • Development process • Description of the assessment • Validity and reliability

The Assessment in Context

Implicit Theory of Action Improved Increased Increased Teacher Professional Classroom K-12 Student Knowledge for Development Practice in Achievement Teaching Science Science in Science Science Curriculum Materials

Implicit Theory of Action

Content Areas • Force and m otion • Processes that shape Earth (Plate Tectonics) • Flow of matter and energy in living systems

Development Process

Development of Teacher Assessments 1. Clarify the science content. 2. Identify student and teacher thinking about the benchmark. 3. Write open-ended items for teachers. 4. Write multiple choice items for teachers. 5. Interview teachers. 6. Pilot and field test items. 7. Create assessment.

Clarifying the Science Content • Benchmark statement “An unbalanced force acting on an object changes its speed or direction of motion, or both. 4F/ M3a” (Project 2061). • “Sub-idea” examples – A force is a push or pull interaction between two objects, and has both magnitude and direction. – If an object is moving faster and faster, then there is a net force acting on the object in the same direction as the motion.

Expert Review • Content clarification • Assessment questions – Before pilot—content accuracy – Before field test—content accuracy – Final form—content accuracy and domain coverage

Psychometric Analysis of Pilot and Field Test Data • Dimensionality analysis (factor analysis) • Differential item functioning (DIF) analysis • Item response theory (IRT) analysis • Scale construction using 2-parameter IRT

Description of the Assessment

In a unit on force and motion, students have been pushing small carts across their tables to the right and observing the motion. The teacher asks the students to draw a diagram showing all the horizontal forces on a cart once it leaves the student's hand and is rolling across the table. (A) Which one of the following force diagrams is correct? 

Common Features of All Items • All are multiple choice • All are keyed to a specific idea  • All are set in the context of work that teachers do 

Target Ideas for Cart Item 1. A force is a push or pull interaction between two objects, and has both strength and direction. 2. The force of friction acts to oppose an object’s motion. 

Other Types of Items

A teacher asks his students, “Imagine you give a block a quick shove so that it slides across a level floor. The block goes slower and slower as it gets farther away from you. In which direction is the net force acting on the block once it leaves your hand?” Most students reply that the net force is acting in the forward direction but getting smaller because of friction. Which of the following is an accurate assessment of the students’ response? (D)  A . The students are correctly combining all of the forces acting on the block. B . The students incorrectly think that forces have strength as well as direction. C . The students correctly think that friction uses up other forces. D . The students incorrectly think that after the block has been pushed it has a force in it that keeps it in motion.

Level 2 Item Features • Address teachers’ ability to analyze student thinking using science content knowledge  • Cannot be answered without content knowledge • Only one answer choice is “content- correct” and relevant to the instructional context • Fairly high cognitive load

Common Errors Made With Level 2 Items  • Teachers look for common student thinking rather than the thinking of these students • Teachers look for a correct statement • Teachers try to answer the student item • Teachers look for familiar wording – e.g., “equal and opposite” • Teachers need options that allow them to hold naïve conceptions

Sample Item

The question below is administered to a group of middle school students. A fan is attached to a cart. The fan is turned on and the cart begins to move along the track. Friction and air resistance are sm all enough to be ignored. What will happen to the cart after it starts moving?

If a majority of students predict that the cart will keep moving at a constant speed, which one of the following would be a good next step in the development of these students’ understanding of the effect of a constant non-zero net force on motion? (C)  A. Reinforce the concept by asking a similar question but in a different context. B. Have students experiment by adding different size masses to the cart. C. Demonstrate for students what happens when a constant non-zero net force is applied. D. Introduce students to the concept of how a changing force affects an object’s acceleration.

Level 3 Item Features  • Address teachers’ ability to make instructional decisions using science content knowledge • Cannot be answered without content knowledge • Only one answer choice is “content- correct” and relevant to the instructional context • High cognitive load

Common Errors Made With Level 3 Items  • Teachers see all activities/ questions as “best” – Lack of content knowledge – High cognitive load • Context is important – Focus on logistics – Unfamiliar scenario/ equipment • Teacher beliefs

In Summary – 3 Kinds of Items • Knowledge of content • Using content knowledge to analyze student thinking • Using content knowledge to make instructional decisions

Validity and Reliability

Reliability • IRT reliability 0.87 • Cronbach’s alpha 0.84

Validity Study Research Questions 1. Do students score higher on the assessment following instruction, and if so, what is the magnitude of the change? 2. Assuming a significant change in student scores exists, is there a relationship between this change and scores on the teacher assessment?

The Sample of Teachers and Students • State-funded MSP administered the force and motion teacher assessment to the 60 high school teachers. • Same teachers were invited to administer the student assessment. • 25 teachers returned completed materials, representing 1,730 students.

Results • Post-test student Averge Force and Motion Student Assessment Scores assessment scores 80 were significantly 70 different than the Percent Correct 6 7 .5 60 pre-test scores, with 50 4 8 .6 an effect size of 0.84 40 30 standard deviations. 20 10 0 Pre- test Post- test

Student Gains Linked to Teacher Assessment Scores • Statistically significant relationship between teacher knowledge of Force and Motion and the change from pre- to post-test on the student Force and Motion assessment • An effect of 0.19 standard deviations Student Scores by Teacher Content Know ledge 8 0 Percent Correct 7 0 7 1 .6 6 0 6 7 .5 6 3 .5 5 0 4 8 .6 4 8 .6 4 8 .6 Pre-test 4 0 3 0 Post-test 2 0 1 0 0 Below Average Average Teacher Above Average Teacher* Teacher* * Below and above average refer to one standard deviation below or above the average teacher score

In-depth Look at 1 Sub-idea I : If an object has constant speed in a straight line (or zero speed), then there is no net force acting on the object. This can occur either when the forces on the object are balanced, or when there are no forces exerted on the object Related misconception: If an object is moving at constant speed, there must be a constant net force in the direction of motion. Two groups of students to look at: 1. Students whose teacher correctly answered two selected items about sub-idea I 2. Students whose teacher did not correctly answer two selected items about sub-idea I

To see how well students understand the force ideas she has been teaching, a teacher asks her students to write statements describing the net force on a ball and its motion. The diagram shows the ball and all the forces acting on it. Most students in the class write two statements: “Forces in opposite directions alw ays cancel each other out and result in a net force of zero.” and “Since the net force acting on the ball is zero, it m ust not be m oving.” Which one of the following represents the best assessment of these students’ understanding? A . Both statem ents are incorrect . B . Statement 1 is correct and statement 2 is incorrect . C . Statement 1 is incorrect and statement 2 is correct . D . Both statements are correct .