Opportunity versus Challenge: Exploring Usage of Log-File and Process Data in International Large Scale Assessments Assessment in the age of Data Science: the case of interactive items tested in France Reinaldo Dos Santos DEPP, Ministry of Education, France ASSESSMENT IN THE AGE OF DATA SCIENCE: THE CASE OF INTERACTIVE ITEMS TESTED IN FRANCE OPPORTUNITY VERSUS CHALLENGE: EXPLORING USAGE OF LOG-FILE AND PROCESS DATA IN INTERNATIONAL LARGE SCALE ASSESSMENTS
ASSESSMENTS IN FRANCE Rome, 6-7 June 2019 ASSESSMENT IN THE AGE OF DATA SCIENCE: THE CASE OF INTERACTIVE ITEMS TESTED IN FRANCE OPPORTUNITY VERSUS CHALLENGE: EXPLORING USAGE OF LOG-FILE AND PROCESS DATA IN INTERNATIONAL LARGE SCALE ASSESSMENTS
ASSESSMENTS IN FRANCE DIGITAL TRANSITION ■ Middle and high schools : ■ Online platform TAO ■ Elementary schools : ■ Offline app on tablets ASSESSMENT IN THE AGE OF DATA SCIENCE: THE CASE OF INTERACTIVE ITEMS TESTED IN FRANCE OPPORTUNITY VERSUS CHALLENGE: EXPLORING USAGE OF LOG-FILE AND PROCESS DATA IN INTERNATIONAL LARGE SCALE ASSESSMENTS
OPPORTUNITIES INNOVATIVE ASSESSMENTS ■ Examples of interactive items ChatBot Maths Physics Coding ASSESSMENT IN THE AGE OF DATA SCIENCE: THE CASE OF INTERACTIVE ITEMS TESTED IN FRANCE OPPORTUNITY VERSUS CHALLENGE: EXPLORING USAGE OF LOG-FILE AND PROCESS DATA IN INTERNATIONAL LARGE SCALE ASSESSMENTS
DIGITAL ITEMS IN MATHS: EVALUATE« WITH » AND « THROUGH » ■ Evaluate « through » digital assessments : to place the student in a digital environment for the assessment. ■ Computer, tablet, … ■ Evaluate « with » digital assessments : to give the student the opportunity to use the technology in order to solve problems. ■ Calculators, dynamic geometry, interactive items … ■ References: ■ Stacey & Wiliam (2013). ■ Drijvers (soon). ASSESSMENT IN THE AGE OF DATA SCIENCE: THE CASE OF INTERACTIVE ITEMS TESTED IN FRANCE OPPORTUNITY VERSUS CHALLENGE: EXPLORING USAGE OF LOG-FILE AND PROCESS DATA IN INTERNATIONAL LARGE SCALE ASSESSMENTS 5
THE CONCEPT OF FUNCTION IN GRADE 9 : ASSESSMENT IN THE AGE OF DATA SCIENCE: THE CASE OF INTERACTIVE ITEMS TESTED IN FRANCE OPPORTUNITY VERSUS CHALLENGE: EXPLORING USAGE OF LOG-FILE AND PROCESS DATA IN INTERNATIONAL LARGE SCALE ASSESSMENTS
INTERACTION, USE, ADAPTATION ■ Qualitative knowledge of the concept of function, through the simulation of a real life situation. ■ Two possible approaches ■ Operational approach : « input/output » The function is understood as a sequence of values ■ Structural approach : graphical perception The function is understood as a math object with properties ■ References: Sfard (1991), Drijvers (2012) ASSESSMENT IN THE AGE OF DATA SCIENCE: THE CASE OF INTERACTIVE ITEMS TESTED IN FRANCE OPPORTUNITY VERSUS CHALLENGE: EXPLORING USAGE OF LOG-FILE AND PROCESS DATA IN INTERNATIONAL LARGE SCALE ASSESSMENTS
COMPLEX DATA Unstructured (text, images, tweets…) or partially structured (JSON, XML …) data. logs recorded as JSON files 15-20 seconds of interacting with the item create a JSON file with 10 000 lines « Big Data » needed! ASSESSMENT IN THE AGE OF DATA SCIENCE: THE CASE OF INTERACTIVE ITEMS TESTED IN FRANCE OPPORTUNITY VERSUS CHALLENGE: EXPLORING USAGE OF LOG-FILE AND PROCESS DATA IN INTERNATIONAL LARGE SCALE ASSESSMENTS
BIG DATA ARCHITECTURE ■ Hadoop & Spark frameworks ■ Hadoop – free and opensource framework, designed to deal with huge volumes of data, in a distributed environment. ASSESSMENT IN THE AGE OF DATA SCIENCE: THE CASE OF INTERACTIVE ITEMS TESTED IN FRANCE OPPORTUNITY VERSUS CHALLENGE: EXPLORING USAGE OF LOG-FILE AND PROCESS DATA IN INTERNATIONAL LARGE SCALE ASSESSMENTS
WORKFLOW Model OK? ASSESSMENT IN THE AGE OF DATA SCIENCE: THE CASE OF INTERACTIVE ITEMS TESTED IN FRANCE OPPORTUNITY VERSUS CHALLENGE: EXPLORING USAGE OF LOG-FILE AND PROCESS DATA IN INTERNATIONAL LARGE SCALE ASSESSMENTS 10
DATA PREPARATION Combine the score of the student with the logs Creation of new variables : Distance between the first input and the target First input in a range around the target (300-500) ASSESSMENT IN THE AGE OF DATA SCIENCE: THE CASE OF INTERACTIVE ITEMS TESTED IN FRANCE OPPORTUNITY VERSUS CHALLENGE: EXPLORING USAGE OF LOG-FILE AND PROCESS DATA IN INTERNATIONAL LARGE SCALE ASSESSMENTS 11
CLASSICAL ANALYSIS The duration alone doesn’t discriminate between success or failure at the item ASSESSMENT IN THE AGE OF DATA SCIENCE: THE CASE OF INTERACTIVE ITEMS TESTED IN FRANCE OPPORTUNITY VERSUS CHALLENGE: EXPLORING USAGE OF LOG-FILE AND PROCESS DATA IN INTERNATIONAL LARGE SCALE ASSESSMENTS 12
CLASSICAL ANALYSIS Still no criteria able to discriminate the populations ASSESSMENT IN THE AGE OF DATA SCIENCE: THE CASE OF INTERACTIVE ITEMS TESTED IN FRANCE OPPORTUNITY VERSUS CHALLENGE: EXPLORING USAGE OF LOG-FILE AND PROCESS DATA IN INTERNATIONAL LARGE SCALE ASSESSMENTS 13
BIG DATA + MACHINE LEARNING = ASSESSMENT IN THE AGE OF DATA SCIENCE: THE CASE OF INTERACTIVE ITEMS TESTED IN FRANCE OPPORTUNITY VERSUS CHALLENGE: EXPLORING USAGE OF LOG-FILE AND PROCESS DATA IN INTERNATIONAL LARGE SCALE ASSESSMENTS 14
MACHINE LEARNING: A TOOLBOX ASSESSMENT IN THE AGE OF DATA SCIENCE: THE CASE OF INTERACTIVE ITEMS TESTED IN FRANCE OPPORTUNITY VERSUS CHALLENGE: EXPLORING USAGE OF LOG-FILE AND PROCESS DATA IN INTERNATIONAL LARGE SCALE ASSESSMENTS 15
UNSUPERVISED LEARNING: CLUSTERING Data partitioning or unsupervised classification consists in splitting a population into homogenic groups. It is often used without prior hypothesis. ASSESSMENT IN THE AGE OF DATA SCIENCE: THE CASE OF INTERACTIVE ITEMS TESTED IN FRANCE OPPORTUNITY VERSUS CHALLENGE: EXPLORING USAGE OF LOG-FILE AND PROCESS DATA IN INTERNATIONAL LARGE SCALE ASSESSMENTS
DBSCAN ■ DBSCAN tends to regroup in the same clusters points that are in the « neighbourhood » of other points of the same cluster. Simple algorithm Systematically convergent Doesn’t need a prior definition of the number of clusters Bad if the clusters’ densities are too different ASSESSMENT IN THE AGE OF DATA SCIENCE: THE CASE OF INTERACTIVE ITEMS TESTED IN FRANCE OPPORTUNITY VERSUS CHALLENGE: EXPLORING USAGE OF LOG-FILE AND PROCESS DATA IN INTERNATIONAL LARGE SCALE ASSESSMENTS 17
DBSCAN Clear separation between the « success » cluster and the « failure » cluster No explanatory variable other than the score (indeed) Can we predict the score depending on other variables? ASSESSMENT IN THE AGE OF DATA SCIENCE: THE CASE OF INTERACTIVE ITEMS TESTED IN FRANCE OPPORTUNITY VERSUS CHALLENGE: EXPLORING USAGE OF LOG-FILE AND PROCESS DATA IN INTERNATIONAL LARGE SCALE ASSESSMENTS 18
SUPERVISED LEARNING: A CLASSIFICATION Supervised learning fits with already labelled data (here, the student’s score at the item). We are trying to predict the label of each individual, through the use of a classification model. Therefore, we split the population between a training sample, with which we will build our model, and a test sample, on which we will test the model’s solidity. ASSESSMENT IN THE AGE OF DATA SCIENCE: THE CASE OF INTERACTIVE ITEMS TESTED IN FRANCE OPPORTUNITY VERSUS CHALLENGE: EXPLORING USAGE OF LOG-FILE AND PROCESS DATA IN INTERNATIONAL LARGE SCALE ASSESSMENTS
RANDOM FORESTS Random forests are the generalization of the decision tree algorithm. The main problem with the decision tree is that the final tree is strongly dependent on the order in which the variables are picked (tree vs leaf). ASSESSMENT IN THE AGE OF DATA SCIENCE: THE CASE OF INTERACTIVE ITEMS TESTED IN FRANCE OPPORTUNITY VERSUS CHALLENGE: EXPLORING USAGE OF LOG-FILE AND PROCESS DATA IN INTERNATIONAL LARGE SCALE ASSESSMENTS
RANDOM FORESTS • For each tree, we pick a random sample of variables. • Each tree is independently trained. • The forest is built through the majority vote of each tree. ASSESSMENT IN THE AGE OF DATA SCIENCE: THE CASE OF INTERACTIVE ITEMS TESTED IN FRANCE OPPORTUNITY VERSUS CHALLENGE: EXPLORING USAGE OF LOG-FILE AND PROCESS DATA IN INTERNATIONAL LARGE SCALE ASSESSMENTS
RANDOM FORESTS Model unable to correctly predict the score with these variables. Feature engineering needed! ASSESSMENT IN THE AGE OF DATA SCIENCE: THE CASE OF INTERACTIVE ITEMS TESTED IN FRANCE OPPORTUNITY VERSUS CHALLENGE: EXPLORING USAGE OF LOG-FILE AND PROCESS DATA IN INTERNATIONAL LARGE SCALE ASSESSMENTS 22
FEATURE ENGINEERING Didactic analysis: Addition of new variables : Gap between the second In a structural approach, students should graphically input and the target Gap between the last detect a narrow zone around the intersection of the curbs, input and the target Standard deviation of the and focus one’s tries into this target zone. input ASSESSMENT IN THE AGE OF DATA SCIENCE: THE CASE OF INTERACTIVE ITEMS TESTED IN FRANCE OPPORTUNITY VERSUS CHALLENGE: EXPLORING USAGE OF LOG-FILE AND PROCESS DATA IN INTERNATIONAL LARGE SCALE ASSESSMENTS 23
EVOLUTION OF THE MODEL The new variables make the random forests more predictive. ASSESSMENT IN THE AGE OF DATA SCIENCE: THE CASE OF INTERACTIVE ITEMS TESTED IN FRANCE OPPORTUNITY VERSUS CHALLENGE: EXPLORING USAGE OF LOG-FILE AND PROCESS DATA IN INTERNATIONAL LARGE SCALE ASSESSMENTS 24
Recommend
More recommend
