Global Alliance to Monitor Learning: Update on Progress Silvia - PowerPoint PPT Presentation

Global Alliance to Monitor Learning: Update on Progress Silvia Montoya, UIS Director GAML4 November 2017 – Madrid, Spain

OUTLINE • Activities of the GAML Secretariat • Progress for each of indicators: o Indicator 4.1.1; o Indicator 4.2.1; o Indicators 4.4.1 and 4.4.2; o Indicator 4.6.1; and o Indicators 4.7.4 and 4.7.5 2

The GAML Secretariat Knowledge Production Technical Commissioning Data Collection partners papers GAML Secretariat Task Force SPC virtual Blogs GAML website meetings meetings Coordination support to Task Communication and Outreach Forces Task Force expert UNESTEAMS meetings 3

Progress in Indicators: 4.1.1 Indicator 4.1.1 Tier III  Multi-tier indicator Proportion of children and young people: (a) in grades 2/3; (b) at the end of primary; and (c) at the end of lower secondary achieving at least a minimum proficiency level in (i) reading and (ii) mathematics. Tier II  Methodological requirements: • Development plan to allow for cross-national 4.1.1.a comparability • Expanding comparability to express all assessments in: 4.1.1.b • Same reporting scale, and • Minimum standards of quality 4

Global Alliance to Monitor Learning: Update on Progress Global consultation Coding Scheme Conceptual Framework Determine common Reference List content UIS & IBE Mapping PLDs Mapping of CNAs Indicator 4.1.1 Upgrading 4.1.1.a UIS Reporting Scales Methodological Reporting Framework Framework UIS Performance Setting benchmarks Metrics on UIS RS General Background Data Alignment Questionnaire for indicator 4.1.1 Social Moderation Interim Reporting 5

Research Activities • Exploring the Commonality and Difference of Regional and International Assessments UIS carried out the following research • A Review of the use of cross-national assessments data in activities for sound educational practice methodological • Mind the Gap: Proposal for a Standardized Measure for decisions to be SDG 4 – Education 2030 Agenda taken… • The Methodology for a Global Composite Indicator for Education: Counting the Number of Children Not Learning • More Than One-Half of Children and Adolescents Are Not Learning Worldwide • Analysis of results of 2017 Survey of Cross-national Assessments • Background Questionnaires • Contribution of Large Scale Assessments to the Monitoring of SDG4 6

Global Alliance to Monitor Learning: Update on Progress Post Estimate Anchoring and Children not Learning Reporting Progression Anchoring Completion Results methodology OOSCi Methodology to provide globally comparable data 617m children and for the proportion of students reaching MPL in adolescents not learning reading and math Methodology to calculate “Children not learning”  quantity and quality in measurement 7

Understanding the Funding of Learning Assessments UIS & GPE Participation in Cross-national assessments Supporting Have you approached non- What is the cost of participating in institutions governmental institutions? an assessment? What are the aspects that need to Is financing secured? be funded? Countries Are you considering non- Do you need technical governmental financial assistance? assistance? Assessment agencies 8

Progress in Data Collection Catalogue of Learning Assessments 2.0 Module 1 Module 2 Module 3 mapping data evaluation instrument collection tool instrument national data on learning characteristics evaluate the robustness outcomes of learning of assessment systems assessments in + + countries national definitions of identify capacity building proficiency levels needs 9

Framework for Task Forces Work Phase What it addresses Main Components • Assessment framework (cognitive, non-cognitive, Conceptual What and who to assess? and contextual) Framework • Target population • Test design • Sampling frame • Operational design Methodological How to assess? • Data generation Framework • Data analysis (e.g. classical or Item Response Theory) • Contextual information • Defining scales • Benchmarking Reporting • Defining progress How to report? Framework (longitudinal equating) • Definition of an interim 10 reporting strategy

Progress in Indicators: 4.2.1 GAML Task Force 4.2 IAEG-ECD Advisory role To support this work, UIS Definition of interim • commissioned 2 papers: Measurement Options for Domains of • • measurement; holistic Development of Interim approach Sustainable Development measurement Goal Indicator 4.2.1, and strategy Developmentally on track • Key Questions on the • Domains of Measurement Annotations • for SDG 4.2.1 11

Global Alliance to Monitor Learning: Update on Progress Indicators 4.4.1 and 4.4.2 What is a globally • Global Competency Framework of Reference on agreed definition of ICT digital literacy skills – Center for Information and digital literacy Technology in Education, University of Hong skills? Kong • DigComp as a base and review cross- national and national competence, curriculum and assessment frameworks of ICT and digital literacy skills • Online consultation • Measurement strategy by Task Force chair 12

Global Alliance to Monitor Learning: Update on Progress • Classified inventory of literacy assessments Indicator 4.6.1 • Four criteria are used to describe What is a concrete assessment/survey on literacy: definition of “fixed proficiency levels” & o Definition invokes continuum “functional literacy and numeracy”? o Assessment covers full range of skills o Statistical methods confirm LAMP psychometric stability o Statistical methods support comparison Reading of Reading of continuous non- • A glance of existing assessments grouped text continuous into four: text Numeracy o Direct assessments: international vs skills national o Indirect assessments: international vs national 13

Global Alliance to Monitor Learning: Update on Progress Indicators 4.7.4 & 4.7.5 What is “adequate • Cross-study on the results of ICCS 2016 & understanding”? 1974 6th consultation • Study to map ESD measurement tool What is “proficiency”? • Measurement strategy by Task Force What are “ issues related to sustainable development and global Conceptual Reporting Operational citizenship”? issues thresholds Framework Work Programme 14

Summary of Process Tools/ Progress to Expected date Steps Actions Output Responsible Process Date of Completion Objectives Global Reporting UIS reporting Doc Databases UIS 2017 protocol Lit National Improvement Catalogue of Plan if needed Learning Pilot Modules 1 Mapping Assessment UIS/Partners and 2 2018 characteristics (modules 1 undergoing and 2) Data CN finished alignment UIS/technical Mapping results Tool in 2018 recording Partners development tool (DART) System- Wide Analysis of Concept Note Concept Note ACER 2018 Assessment Tool for discussion Practices (SWAAP) 15

Summary of Process (cont’d) Expected Tools/ Steps Actions Output Responsible Progress to Date date of Process Completion UIS Reporting Scale 1. 1. Contents Mapping of UIS/IBE mapping of Mapping for 2018 Construction standards CNA CNAFs Math NAFs and of UIS IEA/Regional Work CNA finalized Reporting would round up Reading Scale process Undergoing UIS proposed UIS Draft UIS/IBE mapping of Math almost 2018 definition on GFFR NAFs finalized Concepts Reading in Progress Mapping pf UIS Draft PL UIS/IBE mapping of CN proposal 2018 CNA PLs and PLD NAFs Guidelines and PLDs protocols for working 2. Number and Mapping and UIS Draft UIS led with CN proposal 2018 name of Prof experts levels experts/countries Guidelines and Levels judgment protocols for working 3. Policy UIS Proposal UIS Draft UIS led with CN proposal 2018 Proficiency level PLDs experts/countries Guidelines and descriptors protocols for working 4. Full definition of UIS proposal UIS UIS led with CN proposal 2018 the Performance Complete experts/countries Guidelines and Standards description protocols for 16 of PLDs working

Summary of Process (cont’d) Expected Tools/ Progress to Steps Actions Output Responsible date of Process Date Completion Data alignment for reporting 2. Socially- 1. Mapping CN proposal Moderated alignment of Protocol Degree of UIS/Experts/technical Guidelines 2018 Alignment Policy Level concept note Alignment partners and protocols Descriptors for working 2. Set socially Protocol for CN proposal moderated Protocol mapping UIS/Experts/technical Guidelines 2018 Performance concept note Experts/ partners and protocols levels Countries for working 3. CN proposal Psychometric Protocol / Guidelines 1. Alignment Experts UIS/Partners 2018 Alignment Meeting and protocols for working Protocol / Field Degree of 2. Concordance UIS/Partners Guidelines 2020 Work Alignment 3. Test and/or Protocol / Field Degree of Guidelines UIS/Partners 2020 item based linking Work Alignment and protocols 4. Pair wise comparison? 4. Procedural 1. Define Guidelines Alignment minimum set of a. Sampling Protocol UIS/Partners and protocols processes that for working grant quality 2018 b. Rate of Protocol UIS/Partners 17 response c. Translation Protocol UIS/Partners

Thank you! Silvia Montoya, Director, UNESCO Institute for Statistics @Montoya_sil Learn more: http://uis.unesco.org/ @UNESCOstat