KS2 Mathematics Content Validation Study Paul E. Newton 5 December 2017
Chronology of my involvement Test Preparation (IOE-Nuffield) Content Validation (Ofqual) ■ November 2013 □ joint applicant on Melanie’s successful bid to Nuffield ■ October 2014 ■ October 2014 □ switched role to ‘adviser’ □ joined Ofqual ■ Early 2016 □ began to plan content validation study in advance of new KS2 tests
Why was Ofqual interested in the new KS2 tests? Ofqual has two statutory objectives in relation to national assessments: 1. Assessments standards □ to promote their development and implementation (NAs are sufficiently valid ) 2. Public confidence □ to promote public confidence in them (NAs are trusted )
The Ofqual study had a different focus from the IOE-Nuffield study Test Preparation Content Validation Teachers’ approaches to preparing STA’s approach to domain sampling in pupils for KS2 mathematics tests the new suite of mathematics tests ■ perceived patterns, over time , in ■ actual patterns, during 2016 , in test test questions set; and questions set; and ■ teachers’ responses to them ■ their relation to the new NC
NC Framework = point of reference for STA’s approach to domain sampling Derives from DfE’s National Curriculum Mediated by STA’s Test Framework
NC Framework = point of reference for STA’s approach to domain sampling General aims Specific teaching (learning) objectives
Test Framework re-represents NC’s 173 bullet points as 195 requirements
Test Framework specifies two domains (for the purpose of sampling) Content domain with 9 ‘strands’ Cognitive domain with 5 ‘strands’ (straight from the NC) (developed by STA for the new maths tests) 1. Depth of understanding Number, Ratio and Algebra 1. Number (N) – number and place value 2. Computational complexity 2. Number (C) – addition & subtraction, 3. Spatial reasoning & data interpretation multiplication & division 3. Number (F) – fractions 4. Response strategy 4. Ratio and proportion (R) 5. Algebra (A) Measurement, Geometry and Statistics 6. Measurement (M) 7. Geometry (G) – properties of shapes 8. Geometry (P) – position and direction 9. Statistics (S)
Questions differ in terms of the subject content they test Calculation Geometry – properties and shapes It’s reasonably straightforward to distinguish between, say, a calculation question and a geometry question.
Questions also differ in terms of the cognitive demands they make Calculation Calculation Although it’s far less obvious what kind of features distinguish a less demanding calculation question from a more demanding one.
Rationale underlying the STA’s cognitive domain ■ Questions differ in terms of the nature , and/or the level , of the cognitive challenges (i.e. demands) that they pose. ■ It’s just as important that the test samples adequately from the cognitive domain as from the content domain. ■ The four cognitive domain strands aim to capture the most important kinds of cognitive demands (for KS2 maths questions).
Depth of Understanding (as defined by STA) ■ This strand is used to assess the demand associated with recalling facts and using procedures to solve problems . □ Questions requiring less depth of understanding require simple procedural knowledge, such as the quick and accurate recall of mathematical facts or the application of a single procedure to solve a problem. □ At intermediate levels of demand, a question may require the interpretation of a problem or application of facts and procedures. However, the component parts of these questions are simple and the links between the parts and processes are clear. □ At a high level of demand, a greater depth of understanding is expected. Questions may require that facts and procedures will need to be used flexibly and creatively to find a solution to the problem.
Depth of Understanding Lowest Level of (DoU) Demand High Level of (DoU) Demand LOWEST = SOME (not none)
Test Framework sampling requirements for the cognitive domain
Test Framework sampling requirements for the content domain
Principal Research Questions 1. Do items from the KS2 test embody content and cognitive demands that are consistent with the national curriculum ( item relevance )? 2. Does the KS2 test effectively sample those content and cognitive demands adequately ( test representativeness )? … in terms of how STA experts rated the ■ content demands of each item from each of the three papers ■ cognitive demands of each item (for each strand) from each of the three papers … in terms of how 12 independent experts rated the ■ content demands of each item from each of the three papers ■ cognitive demands of each item (for each strand) from Papers 2 and 3
Test Framework sampling requirements for the cognitive domain
Test Framework sampling requirements for the content domain
What about lower level sampling expectations for the content domain?
Independent SMEs’ views on content demands
What the results looked like (e.g. Paper 3)
Independent SMEs’ views on cognitive demands We’d like you to rate each item (from Paper 2 and 3) ■ using the STA definition for each of the four cognitive demand strands □ DO your best to understand exactly what STA means by demand (for each strand) ■ using your own sense of how demanding the item is (in terms of each strand) □ DON’T necessarily try to ‘replicate’ STA application of 4-point-scale (for each strand) □ USE scale to indicate your opinion of the level of demand of the item (for each strand)
What the results looked like (Paper 2 and Paper 3)
What the results looked like (Paper 2 and Paper 3)
Conclusions 1. In terms of how the STA (i) represented domain sampling requirements for the mathematics curriculum, and (ii) judged the content and cognitive demands of items from the 2016 test a. the items were all relevant to the NC, and b. the test overall was representative of the NC 2. Our SMEs largely agreed with how STA judged the content and cognitive demands of items from the 2016 test, which a. provided additional support for conclusion 1 , and b. supported the plausibility of the innovative cognitive domain strands STA’s approach to domain sampling compares favourably with approaches adopted for similar tests, internationally.
However… Still room for debate over the ‘proper’ interpretation of domain sampling requirements for the mathematics curriculum ■ STA’s atomistic approach to domain sampling tends to focus attention more on the NC’s (bullet point) teaching requirements than on its Aims ■ STA’s guidelines require only a small number of high cognitive demand items
Consider how the depth of understanding strand relates to problem solving The Test Framework distinguishes The Aims of the NC for mathematics between rating points 3 and 4 on the include: depth of understanding strand: ■ can solve problems by applying their 3 use facts and procedures to solve mathematics to a variety of routine more complex problems and non-routine problems with increasing sophistication, including 4 understand and use facts and breaking down problems into a procedures creatively to solve series of simpler steps and complex or unfamiliar problems persevering in seeking solutions. (Bear in mind that STA rated 5/110 marks from the 2016 test at DoU 4)
For full details of this research…
Recommend
More recommend
