Overview eat - History 1 eat: An R Package for Automation of Data - PowerPoint PPT Presentation

Overview eat - History 1 eat: An R Package for Automation of Data Preparation The Institute for Educational Quality Improvement and IRT Modeling ACER ConQuest The Idea eat - Concept 2 Karoline Sachse, Martin Hecht, Sebastian Weirich, Overview Nicole Haag, Malte Jansen, Sebastian Wurster, Data Christiane Penk, Anna Lenski, Thilo Siegle eat - Examples 3 Institute for Educational Quality Improvement Humboldt-University, Berlin Data Preparation Unidimensional 1PL model with automateModels Grouping options in automateModels February 10, 2012 automateModels Discussion 4 Outlook Psychoco, Innsbruck February 10, 2012 1 / 26 Psychoco, Innsbruck February 10, 2012 2 / 26 eat - History The Institute for Educational Quality Improvement eat - History ACER ConQuest The Institute ConQuest Commercial Software developed by ACER (Wu, Adams & Wilson, 1997) Independent research and test institute founded by the 16 federal Major scaling tool of the Organisation for Economic Co-operation and states in 2004 Development’s Programme for International Student Assessment Nationwide Educational Standards Assessments in German, the first (PISA) foreign language, Mathematics and Science which allow comparison Fits a large number of different item response models of federal states ( N ≈ 30 , 000) Rasch, partial credit, rating scale, facets, ... Assessment tests in German, Mathematics and the first foreign Latent regression language in the 8th grade at secondary school (once a year) Multidimensionality Assessment tests in German and Mathematics in the 3rd grade at Estimation primary school (once a year) Marginal Maximum Likelihood Gaussian quadrature/ Monte Carlo approximations Person parameter estimation: EAP, MLE, WLE, Plausible values Psychoco, Innsbruck February 10, 2012 3 / 26 Psychoco, Innsbruck February 10, 2012 4 / 26

eat - History The Idea eat - Concept Overview Automation of Data Preparation and Analysis Implemented Modules automate data preparation read in & check SPSS-files 1 merge data frames (booklets) 2 recode & dichotomize data 3 automate IRT calibration write ConQuest syntax, generate appropriate data input 1 execute ConQuest 2 read in ConQuest output 3 facilitate reporting write out results (graphics, tables, ...) 1 ⇒ ”eat” (”Educational Assessment Tools”) Psychoco, Innsbruck February 10, 2012 5 / 26 Psychoco, Innsbruck February 10, 2012 6 / 26 eat - Concept Overview eat - Concept Data Wrapping ConQuest Typical Items Psychoco, Innsbruck February 10, 2012 7 / 26 Psychoco, Innsbruck February 10, 2012 8 / 26

eat - Concept Data eat - Concept Data Typical Items - Scores Data Structure Psychoco, Innsbruck February 10, 2012 9 / 26 Psychoco, Innsbruck February 10, 2012 10 / 26 eat - Examples Data Preparation eat - Examples Data Preparation automateDataPreparation Data Preparation Input dataset <- automateDataPreparation ( inputList = inputList, path = path, loadSav = TRUE, checkData = TRUE, mergeData = TRUE, recodeData = TRUE, aggregateData = TRUE, scoreData = TRUE, writeSpss = TRUE ) Psychoco, Innsbruck February 10, 2012 11 / 26 Psychoco, Innsbruck February 10, 2012 12 / 26

eat - Examples Data Preparation eat - Examples Unidimensional 1PL model with automateModels Data Preparation Logfile Simple Unidimensional 1PL Model results01 <- automateModels( dataset = dataset , folder = folder ) Psychoco, Innsbruck February 10, 2012 13 / 26 Psychoco, Innsbruck February 10, 2012 14 / 26 eat - Examples Unidimensional 1PL model with automateModels eat - Examples Unidimensional 1PL model with automateModels ConQuest Dataset & Label File Creation ConQuest Syntax Creation Psychoco, Innsbruck February 10, 2012 15 / 26 Psychoco, Innsbruck February 10, 2012 16 / 26

eat - Examples Unidimensional 1PL model with automateModels eat - Examples Unidimensional 1PL model with automateModels ConQuest Run ConQuest Output Files ConQuest runs due to automatic creation and execution of batch files Item parameter estimates .shw, .itn, ... Person parameter estimates .wle, .mle, .eap, .pvl, ... ⇒ Many different output files Psychoco, Innsbruck February 10, 2012 17 / 26 Psychoco, Innsbruck February 10, 2012 18 / 26 eat - Examples Unidimensional 1PL model with automateModels eat - Examples Unidimensional 1PL model with automateModels ConQuest Item Parameter Output ConQuest Person Parameter Output Psychoco, Innsbruck February 10, 2012 19 / 26 Psychoco, Innsbruck February 10, 2012 20 / 26

eat - Examples Unidimensional 1PL model with automateModels eat - Examples Unidimensional 1PL model with automateModels eat Reporting eat Log Item parameter estimates all objects (dataset, item.grouping, ...) will be archived into an .RData file an INFO file will be created Person parameter estimates Psychoco, Innsbruck February 10, 2012 21 / 26 Psychoco, Innsbruck February 10, 2012 22 / 26 eat - Examples Grouping options in automateModels eat - Examples Grouping options in automateModels Multidimensional vs. Unidimensional Analysis Person groups & weights results02 <- automateModels( dataset = dataset , id = "id" , folder = folder , item.grouping = item.grouping , select.item.group = c ( "ER" , "EL" ) ) dataset <- cbind ( dataset , "weight1" = as.character(sample(c(0.8, 1, 1.2), nrow(dataset), replace=TRUE)), "weight2" = as.character(sample(c(1), results03 <- automateModels( dataset = dataset , id = "id" , folder = folder , nrow(dataset), replace=TRUE)), stringsAsFactors = FALSE ) item.grouping = item.grouping , select.item.group = c ( "ER" , "EL" ) , cross="all") results04 <- automateModels( dataset = dataset, folder = folder context.vars = c ( "weight1" , "weight2" ) , item.grouping = item.grouping , select.item.group = "ER" , person.grouping = person.grouping , select.person.group = list ( "gr.9" , "gr.10" ) , weight = list ( "weight1" , "weight2" ) ) Psychoco, Innsbruck February 10, 2012 23 / 26 Psychoco, Innsbruck February 10, 2012 24 / 26

eat - Examples automateModels Discussion Outlook automateModels – Overview Thank you automateModels(dataset, id = NULL, context.vars = NULL, items = NULL, item.grouping = NULL, select.item.group = NULL, Thank you for your attention! person.grouping.vars = NULL, person.grouping.vars.include.all = FALSE, person.grouping = NULL, select.person.group = NULL, http://r-forge.r-project.org/eat additional.item.props = NULL, folder, overwrite.folder = TRUE, analyse.name.prefix = NULL, analyse.name = NULL, eat-commits@lists.r-forge.r-project.org analyse.name.elements = NULL, data.name = NULL, m.model = NULL, software = NULL, dif = NULL, weight = NULL, anchor = NULL, Special thanks to regression = NULL, adjust.for.regression = FALSE, q3 = FALSE, missing.rule = NULL, cross = NULL, subfolder.order = NULL, Alexander Robitzsch (Measurement Statistician, bifie) subfolder.mode = NULL, additionalSubFolder = NULL, run.mode = NULL, Martin Mechtel (IT Director, IQB) n.batches = NULL, run.timeout = 1440, run.status.refresh = 0.2, email = NULL, smtpServer = NULL, write.txt.dataset = FALSE, delete.folder.countdown = 5, conquestParameters = NULL ) Psychoco, Innsbruck February 10, 2012 25 / 26 Psychoco, Innsbruck February 10, 2012 26 / 26