A new statistical method for detecting Differential Item Functioning in the Rasch-Model Julia Kopf Achim Zeileis Carolin Strobl LMU
Gist of this presentation Main idea: Method to detect parameter instability in the Rasch-model Usage of model-based recursive partitioning algorithm Application of the method to detect DIF 2/12
Usage of Rasch-Trees Surprising result: Higher general knowledge in Rhineland-Palatinate comparing to other German Federal states 3/12
Usage of Rasch-Trees Surprising result: Higher general knowledge in Rhineland-Palatinate comparing to other German Federal states Possible reason: Differential Item Functioning in the Rasch-Model 3/12
Usage of Rasch-Trees Surprising result: Higher general knowledge in Rhineland-Palatinate comparing to other German Federal states Possible reason: Differential Item Functioning in the Rasch-Model ● Politics 1 ● History A_levels ● Economics Culture ● ● Natural Science other {Hessen, Rheinl.−Pfalz} 3 A_levels Hessen Rheinl.−Pfalz Node 2 (n = 8321) Node 4 (n = 721) Node 5 (n = 400) 4.03 4.03 4.03 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● −2.26 −2.26 −2.26 1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 3/12
Usage of Rasch-Trees Surprising result: Higher general knowledge in Rhineland-Palatinate comparing to other German Federal states Possible reason: Differential Item Functioning in the Rasch-Model ● Politics 1 ● History A_levels ● Economics Culture ● ● Natural Science other {Hessen, Rheinl.−Pfalz} 3 A_levels Hessen Rheinl.−Pfalz Node 2 (n = 8321) Node 4 (n = 721) Node 5 (n = 400) 4.03 4.03 4.03 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● −2.26 −2.26 −2.26 1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 Item 4: Find Hesse on the German map! Item 5: What’s the capital of Rhineland-Palatinate? 3/12
Usage of Rasch-Trees Surprising result: Higher general knowledge in Rhineland-Palatinate comparing to other German Federal states Possible reason: Differential Item Functioning in the Rasch-Model ● Politics 1 ● History A_levels ● Economics Culture ● ● Natural Science other {Hessen, Rheinl.−Pfalz} 3 A_levels Hessen Rheinl.−Pfalz Node 2 (n = 8321) Node 4 (n = 721) Node 5 (n = 400) 4.03 4.03 4.03 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● −2.26 −2.26 −2.26 1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 Item 4: Find Hesse on the German map! Item 5: What’s the capital of Rhineland-Palatinate? Obtained result: ⇒ The questions in the survey do not lead to fair comparisons . 3/12
Overview: The Rasch-model Objective of the Rasch-model: Measurement of latent variables Examples: Obtain at least interval scaled Intelligence and attainment tests person parameters Extensions: These are monotone 2-pl (Birnbaum), 3-pl models transformation of raw scores Essential data: i1 i2 i3 i4 i5 i6 i7 i8 i9 i10 i11 i12 i13 i14 sex* domicile* 1 0 0 0 0 0 1 0 0 0 0 0 0 0 0 man west 5 0 0 0 0 1 1 0 0 0 0 0 0 0 0 woman west 7 0 1 1 0 1 1 0 1 0 0 0 1 0 0 man west 8 1 1 1 0 1 1 0 0 0 1 0 0 0 0 man west 10 1 0 1 0 1 1 0 0 0 1 0 0 0 0 man west 11 1 0 0 0 1 1 0 0 0 1 0 0 0 0 woman west 4/12
Overview: The Rasch-model Assumptions of the Rasch-model (Rasch, 1960): Influence of latent variable Assumptions about Item Characteristic Curves (ICC) Unidimensionality Local stochastic independence Invariance of Item parameters ,,The importance of the property of invariance of item and ability parameters cannot be overstated. This property is the cornerstone of item response theory and makes possible such important applications as equating, item banking, investigation of item bias, and adaptive testing” (Hambleton, Swaminathan and Rogers, 1991: 25). 5/12
Item Characteristic Curves and estimation process Itemcharakteristiken Assumptions about the ICCs: 1.0 Wahrscheinlichkeit für richtige Antwort 0.8 Probability of solving or agreeing as a function of 0.6 latent variable 0.4 item difficulty 0.2 monotone, logistic form Item i1 0.0 Item i2 −10 −5 0 5 10 Latente Dimension Estimation via Conditional Maximum Likelihood (CML): Probability for person i (= 1 . . . , n ) solving item j (= 1 . . . , k ) is: P ( U ij = u ij | θ i , β j ) = exp[( θ i − β j ) · u ij ] 1 + exp( θ i − β j ) , β j denotes the item parameter of item j θ i is the person parameter of individual i u ij ∈ { 0; 1 } symbolizes the answer of person i to item j r i = � k j =1 u ij and s j := � n i =1 u ij 6/12
CML estimation and subject-wise score functions New parametrisation (Fischer und Molenaar, 1995): ξ i = exp( θ i ) and ε j = exp( − β j ) Individual Loglikelihoods: k � Ψ( y i , ε ) = u ij log( ε j ) − log γ r i j =1 with k u ij � � γ r i = ε . j j =1 � k j =1 u ij = r i Elementary symmetric functions (Liou, 1994): γ 0 = 1 γ 1 = ε 1 + ε 2 + . . . + ε k γ 2 = ε 1 · ε 2 + ε 1 · ε 3 + . . . + ε k − 1 · ε k . . . γ k = ε 1 · ε 2 · . . . · ε k Individual Scores: γ ( j ⋆ ) ψ ( y i , ε ⋆ ) = ∂ Ψ( y i , ε ) = u ij ⋆ r i − 1 . − ∂ε ⋆ ε ⋆ γ r i 7/12
Model-based recursive partitioning of Rasch-models Implementation of Achim Zeileis in psychotree The code conversation can be summarized in the following way: 1 Hand-off formula like item1 + item2 + ... + itemk ˜ X1 + X2 + ... + Xl , arguments, data 2 Model class RaschModel including RaschModel.fit 3 Data sanity checks 4 Passing to mob() from package party (Zeileis et al., 2008) Available functions in updated package psychotree : summary() plot() coef() worth() 8/12
Identifying parameter instability Ways of identifying violation of parameter invariance: Graphical model test according to item raw scores and sex According to ability According to sex Difficulty for group with high raw score 3 3 14 14 2 2 Difficulty for men 1 39 1 19 19 29 23 1 32 1 39 29 3 11 13 30 11 89 3 30 8 9 2 28 13 32 2 20 17 17 31 21 25 23 25 41 20 31 6 28 21 5 41 18 5 18 0 6 0 44 34 26 33 26 16 43 43 34 36 44 16 42 36 38 40 33 10 38 40 15 42 15 22 10 24 −1 −1 35 7 24 22 4 37 4 12 37 27 12 35 7 27 45 45 −2 −2 −3 −3 −3 −2 −1 0 1 2 3 −3 −2 −1 0 1 2 3 Difficulty for group low raw score Difficulty for women Likelihood Ratio tests Problem: Which groups may influence the item parameters? 9/12
Recommend
More recommend
