for Testing & Evaluation Tzur Karelitz Sept 2016 Outline - PowerPoint PPT Presentation

of the National Institute for Testing & Evaluation Tzur Karelitz Sept 2016

Outline  NITE: history, structure & objectives  Overview of NITE’s assessment services & activities  A detailed look at the RAMA project: • Analyzing and reporting data from large scale assessments and survey for the Israeli Ministry of Education. 2

NITE: History & Structure  Established in 1981 by a consortium of the universities in Israel in order to centralize admissions testing for applicants to higher education .  NITE is a public, not-for-profit organization , that is supervised by a board of directors (representatives of founding universities).  NITE staff includes about 130 professionals : item writers, statisticians, psychometricians, computer programmers, graphic designers, language editors, and logistic & administrative staff. 3

NITE’s Organizational Structure  CEO: Dr. Anat Ben-Simon  Deputy Director: Dr. Naomi Gafni  Departments  Units • • Test development CAT • • Scoring Non-cognitive tests • • Research Test accommodation • • Operations Computerized LD diagnosis system • IT • Automated text analysis • Finance & administrations • RAMA project 4

NITE: Main Objectives  Provision of assessment services primarily to institutions of higher education,  and also to the educational system (K-12), and other public organizations  Conducting research on admissions, placement, assessment and evaluation in institutions of higher education  Advancement of the fields of measurement, testing and psychometrics in Israel 5

Typical Assessment Service  Design and development of assessments & tools  Test registration  Test administration & accommodations  Scoring and quality control  Reporting of results  Conducting related research 6

Other Assessment Services  Expert item review  Test translation/adaptation  Computerizing P&P tests  Analyzing test results  Conducting evaluation projects  Training professional staff  Consulting organizations in Israel and abroad  Teaching relevant academic courses

Tests in Higher Education Test Admission to - Type ~N PET Universities & colleges P&P / CAT 70,000 Pre-academic prep MEIMAD Online 6,000 schools Assessment MOR/MIRKAM Medical schools 1,700 Centers MITAM Psych. graduate studies P&P 1,300 Test Proficiency in - Type ~N AMIR/AMIRAM English P&P / CAT 45,000 YAEL Hebrew P&P 25,000 And multiple, smaller-scale, admission & proficiency tests to various programs and organizations. 8

The Psychometric Entrance Test (PET)  Admission to higher ed. is based on the mean of PET and school matriculation exams grades (BAGRUT).  PET is a scholastic aptitude test consisting of: • Verbal Reasoning (~60 MC items) • Writing (one task) • Quantitative Reasoning (~50 MC items) • English as a Foreign Language (~ 55 MC items)  PET is a standardized P&P test given 5 times a year • It takes about 3.5 hours to complete • Scores range 300-800, mean 540, SD=110. • Adapted versions for examinees with disabilities  PET is translated into: Arabic, Russian, English, French, and Spanish (and sometimes to Italian & Portuguese) 9

Other projects  MATAL- a comprehensive, standardized, computer-based test battery for the diagnosis of learning disability. • Aiding the provision of test accommodations in higher education.  HLP- computational tools for analyzing and rating Hebrew texts. • NiteRater- automatic essay rating system.  ICAP- an initiative to advance educational measurement and psychometrics in Israel. 10

The RAMA Project: Analyzing and reporting results from large-scale tests and surveys for the Ministry of Education

Outline • Background • Team • Main projects • Tasks • Tools • School Climate and Pedagogical Environment (CPE) surveys • Growth and Effectiveness Measures for Schools (GEMS) • Report production • Challenges and successes 12 17/12/2015 Rama Project

Background • A 5-year, variable-quota contract issued by the MOE, which began in 2012. Expected to be renewed in 2017.  Providing services to “RAMA” – The National Authority for Measurement and Evaluation in Education (a branch of MOE).  Main project cycle is between May and January, peaking in July-October. Test/Survey Development Climate: RAMA Achievement: CET Adminstration , Human Rating & Data Entry TALDOR Cleansing, Analysis, Reporting NITE 13 17/12/2015 Rama Project

The RAMA Team Eran Nethanel DBA Matan Tzur Report Production Manager Psychometrician Valla Analyst And 6 part- time assistants Eliran Evgeny Shaul Analyst Analyst Analyst 14 17/12/2015 Rama Project

Main Projects - 2015 • Growth and Effectiveness Measures for Schools (GEMS): Achievement Tests for 5 th & 8 th grades • First language (Heb/Arab), Math, English, Science & Technology • About 200,000 records per year • Climate and Pedagogical Environment (CPE) Surveys • 5 th – 9 th grade : about 150,000 students and 12,000 teachers per year • 10 th & 11 th grades: about 50,000 students and 8,000 teachers per year  Results for surveys and tests are reported on 4 levels: • Schools (1/3 of the country): about 900 elementary & middle schools, and 300 high schools, per year. • Municipalities : about 100 per year • Districts : 8 every year • National : by language, school type (secular/religious), sub-groups within the Arab population, SES. • Hebrew as a Second Language for Arabic speakers • 7,000 6 th grade students (test and survey) and 600 teachers (survey). • Results reported nationally. 17/12/2015 Rama Project 15

Main Project Tasks • Database design and maintenance • Data cleansing • Dealing with inconsistent, missing, corrupted, inappropriate or duplicate data in surveys, tests and background information • Quality control of surveys prepared by RAMA • Maintenance of item properties in the database • Factor analysis for surveys and tests • Item analysis for surveys and tests Parallel • Scoring using classical scaling and calibration channel • Aggregations and norms • IRT analyses (sometimes) 17/12/2015 Rama Project 16

Main Project Tasks (cont.) • Extraction of historical comparison data • Dealing with special cases • Generation of reports for project control & monitoring • Automatic generation of personalized reports • levels: school, district, municipality, national • Human and automatic quality control of reports • Language and format editing of reports • Writing insights and conclusions based on results • Preparation of CDs and envelopes for mailing • Secondary data analysis (research questions) • Documentation and technical reports 17/12/2015 Rama Project 17

Tools • SQL – Data management, cleansing and history • SAS, SPSS – Data manipulation and analysis • Parallel channel : All the main analyses are performed separately by different analysts using different codes and software, the results are compared, and inconsistencies are resolved. • VBA – automation, quality control, reports’ post - processing • Magic Publisher – Inserting SPSS output into Word • Winsteps – IRT • Word – Reports • Excel – Everything… 17/12/2015 Rama Project 18

School Climate and Pedagogical Environment (CPE) Surveys

Goal of CPE Reports The surveys aim to provide a detailed picture of various aspects of the social climate and pedagogical processes in schools, that are essential for educational quality : satisfaction, relationship within the school community, security and safety, discipline and behavioral issues, emotional-motivational aspects, teaching-learning-assessing processes, value-based education, etc. The reports are meant to help school personnel to set evidence-based goals, and to track, plan and monitor important aspects of schooling such as inter-personal relationships, educational interaction, motivation and educational aptitude among students. 17/12/2015 Rama Project 20

Reported Indicators, 2015 Pedagogy Climate Practices of quality teaching-learning-assessment An overall positive attitude among students towards school Class assignments that promote inquiry learning Close relationship and caring between teachers and students Self-learning strategies Positive relations between students and their peers Receiving feedback to promote learning Involvement in violent incidents School's efforts to encourage social and civic Digital violence via social networks and on the involvement Internet School's efforts to promote tolerance of diversity Verbal violence School trips & tours School’s efforts to encourage a sense of safety Students’ recreational activities Proper behavior of students in the classroom Differentiated instruction at school Teachers’ satisfaction with school Giving feedback to promote learning Involvement of parents in school Teamwork at school Teachers’ lack of a sense of safety Collaborative learning in school Competence, curiosity and interest in learning Teachers’ professional development School's efforts to encourage motivation and curiosity among students 17/12/2015 Rama Project 21