in Low- and Middle-Income Countries to Measure Cognitive Outcomes - PowerPoint PPT Presentation

Overview of Basic Education Instruments Commonly Used in Low- and Middle-Income Countries to Measure Cognitive Outcomes Session 1. Measuring Student Learning Outcomes Marguerite Clarke, PhD The World Bank Group

Overview 1. Cognitive outcomes 2. Common instruments 3. Technical issues 4. Basic steps to keep in mind 2

Learning domains – ‘cognitive’ and ‘non - cognitive’ outcomes Source: Learning Metrics Task Force (2013) http://www.brookings.edu/about/centers/universal-education/learning-metrics-task-force-2 3

Common instruments for measuring cognitive outcomes Pros Cons Best for … EGRA Granular data useful for Not optimized for Targeting classroom informing daily aggregate reporting practices classroom practice NLSA Representative data Technical quality and Tracking policy linked to nationally comparability issues reforms and system relevant standards quality ILSA Comparable data based Typically not Tracking policy on internationally treatment sensitive reforms and system relevant standards quality Bespoke/adapted Optimized for relevance Technical quality and Determining to particular comparability issues effectiveness of context/project specific intervention 4

Common instruments for measuring cognitive outcomes - EGRA Pros Cons Best for … EGRA Granular data useful Not optimized for Targeting classroom for informing daily aggregate reporting practices classroom practice 5

EGRA description • Speeded measure of early reading skills (Grades 1-3) • One-on-one oral delivery • Supply-type items • No background questionnaire • 8 subscale scores (speeded) – Letter name knowledge – Phonemic awareness – Letter sound knowledge – Familiar word reading – Unfamiliar non-word reading https://www.eddataglobal.org/reading/index.cfm – Passage reading and comprehension – Listening comprehension – Dictation 6

Best suited for  Informing daily classroom practice by teachers  Measuring impact of targeted interventions on classroom practices and student learning  Monitoring status and growth at individual and group level over short time frame on specific subscales 7

Common instruments for measuring cognitive outcomes – Bespoke/adapted Pros Cons Best for … Bespoke/adapted Optimized for Technical quality and Determining relevance to particular comparability issues effectiveness of context/project specific intervention 8

Bespoke/adapted general description • Typically based on existing NLSA or ILSA (any grade/age) • Domain-specific knowledge and skills • Publicly released items • Written, group-administered, standardized • Selection- and supply-type items • Background questionnaire • Subscale and overall scores 9

Best suited for  Informing policy decisions  Measuring impact of specific interventions on student learning  Comparisons with other/external groups at item level 10

Example: IEA tests adapted for Benin • Purpose: • Baseline in math and reading for schools in RCT • Sample: • 144 schools, 3,817 students (grade 5) • Instrument development: • Released items and materials from previous IEA • Developed in collaboration with local experts • Mathematics: 30 items, all multiple choice • Reading: 25 items; docs, narrative/expository prose • Background questionnaire 11

12

13

14

15

Technical issues 1. Treatment sensitivity/information value 2. Reliability 3. Comparability 16

Treatment sensitivity (1) 17

Treatment sensitivity (2) 18

Treatment sensitivity (3) 19

Reliability Suitable for EGRA 20

21

Comparability (1) 22

23

Comparability (2) 24

Basic steps to keep in mind Volume 1: Assessing National Achievement Levels in Education Volume 2: Developing Tests and Questionnaires Volume 3: Implementing a National Assessment Volume 4: Analyzing Data from a National Assessment Volume 5: Using the Results of a National Assessment http://go.worldbank.org/M2O1YDQO90

Thank you Marguerite Clarke mclarke2@worldbank.org World Bank Group