Track Student Progress and Inform Continuous Improvement with the - PowerPoint PPT Presentation

Track Student Progress and Inform Continuous Improvement with the NAEP Data Explorer Robert Finnegan, Educational Testing Service Jason Nicholas, Westat Julie Williams, California Department of Education

NAEP Data Explorer History and Overview of the Redesign 2

History of Online Data Analysis Tools • Early 2000s NAEP Data Tool • Two iterations • Allowed users to look up pre- summarized data tables 3

History of Online Data Analysis Tools • Released in 2005 • First iteration of using plausible values to generate summary results • Included user defined significance testing and charting capabilities • Redesigned in 2009 4

Overall Goals for NAEP Reporting 5

Key Goals for the NDE Redesign • Improve user interface for efficient data requests • Save settings and reports • Copy reports across subject and grades • Integrate existing NAEP tools with a search interface • Update charts and graphs • Use NAEP brand styles 6

Select Criteria 7

Select, Combine, View Jurisdictions 8

Select Variables, Create Crosstabs 9

Save Reports, View Saved Reports 10

Copy and Edit Reports 11

Integrated Search Function 12

“Poor interpretations of reports are harmful even if they are based on carefully designed tests.” John Hattie, Educational Measurement: Issues and Practice, Winter 2014, 33(4), pp 34-35.

NAEP Program • Key to understand the NAEP program and its design • Uniquely different than state assessment systems and serves a different purpose • Data from NAEP based on complex samples of schools and students and related interpretations need to account for this aspect

NAEP State Coordinators • Every state has a NAEP State Coordinator who is the “local” expert in all things NAEP – Makes NAEP happen………… • This individual is the key resource for you in accessing NAEP data and conducting analyses • Trained and supported in NAEP analyses and NAEP data tools • Provided valuable feedback and insights into re- design of NDE

NAEP State Coordinators • NAEP Support & Service Center supports the Coordinators with training and assistance • Also TUDA Coordinators in select large urban districts – California • Los Angeles • San Diego • Fresno – Non-California • New York City, Chicago, Dallas, Las Vegas, Miami- Dade……….

NAEP Results and Data • With NAEP state results being available every other year, there are insightful data on student progress available – Trends not available from state assessment data – In addition, contextual information related to the environments in which students learn • NAEP data can provide valuable pathways for state agencies to efficiently “inform” their improvement plans – Provide the spark

Best Practices for Analyzing Data with the NDE • Familiarize yourself with what data are available – Nation’s Report Card website (nationsreportcard.gov) – Survey Questionnaires (search: NAEP questionnaires) • Have a question in mind – Subject, grade, year, jurisdiction • Play with the tool – Don’t get too deep too soon

NDE Hidden Gems • Subscales – Algebra, Geometry, Measurement, Probability & Statistics, Number & Operations – Physical, Earth, Life – Literary, Informational • Multiple statistics – Means & percentiles for scale scores – Percentages for achievement levels – Confidence intervals

NDE Hidden Gems • Cross Tabulation – NSLP participation x teacher tenure • Trend – All states have trend back to 2002 or 2003 – Some back to 1992 • Statistical significance testing – Provides the check

Example • What is impact of teacher’s years of experience on elementary Reading achievement? • Do we know? What is the relationship?

Example • Grade 4 – Reading – National Public – 2017 2 Years or 3 to 10 More than Less Years 10 Years Percent of * * 11% 34% 56% Students Average Scale * * 213 220 223 Score * Significantly different ( p < .05) from More than 10 Years.

Example • Grade 4 – Reading – National Public – 2017 73 30 18 4

Example • Grade 4 – Reading – National Public – 2017 2 Years or More than Less 10 Years Gap Average Scale * 213 223 11 Score 23 * -4 * * Significantly different ( p < .05) from National Public Gap.

Example • Differences by state exist • NAEP cannot infer causality – New teachers don’t teach as well as old teachers? – New teachers get jobs in places with lower achievement? • What other characteristic might be involved? – Parental education – NSLP eligibility – School location – School Title I status

Example • Grade 4 – Reading – National Public – 2017 2 Years or More than Less 10 Years Eligible 13% 53% * * Not Eligible 9% 59% Eligible 204 210 * * Not Eligible 228 238 * Significantly different ( p < .05) from Eligible.

Example • Grade 4 – Reading – National Public – 2017 2 Years or Less Eligible 13% Gap 4% * Not Eligible 9% 11% * -2% * Significantly different ( p < .05) from Eligible or NP gap.