Ordering Individuals with Sum Scores: the Introduction of the - PowerPoint PPT Presentation

The Use of Sum Scores Suffiency Testable Implications Illustrations Future directions Ordering Individuals with Sum Scores: the Introduction of the Nonparametric Rasch Model Robert Zwitser Gunter Maris Cito July 26 th 2013 Robert Zwitser, Gunter Maris Presentation IMPS 2013

The Use of Sum Scores Suffiency Testable Implications Illustrations Future directions Scoring How to score a test? latent variable models How to report test results? parameter estimates versus observed scores Robert Zwitser, Gunter Maris Presentation IMPS 2013

The Use of Sum Scores Suffiency Testable Implications Illustrations Future directions Monotone Latent Variable Models Monotone latent variable models: assuming (at least) Unidimensionality Local Independence Monotonicity In particular: Rasch Model (RM) Monotone Homogeneity Model (MHM) Robert Zwitser, Gunter Maris Presentation IMPS 2013

The Use of Sum Scores Suffiency Testable Implications Illustrations Future directions Monotone Homogeneity Model Stochastic ordering of θ by X + (SOL): (Θ | X + = a ) > st (Θ | X + = b ) , if a > b , 1.0 ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● 0.8 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● which equals ● ● ● ● ● ● ● 0.6 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● F( Θ |X + ) ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● 0.4 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● E ( h (Θ) | X + = a ) > E ( h (Θ) | X + = b ) ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● 0.2 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● X + = a ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● X + = b ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● 0.0 ● ● ● ● ● ● ● for all a > b , and all bounded increasing Θ functions h . Robert Zwitser, Gunter Maris Presentation IMPS 2013

The Use of Sum Scores Suffiency Testable Implications Illustrations Future directions Individual Measurement Consider a test with 3 items: Guttman item two items with P ( X i = 1) = 0 . 5 (e.g., two coins) (Θ | X + = 2) > st (Θ | X + = 1) (Θ | X = [0 , 1 , 1]) < st (Θ | X = [1 , 0 , 0]) Robert Zwitser, Gunter Maris Presentation IMPS 2013

The Use of Sum Scores Suffiency Testable Implications Illustrations Future directions (Ordinal) Sufficiency Sufficiency of statistic H implies that (Θ | X = x 2 ) > st (Θ | X = x 1 ) , if H ( x 2 ) > H ( x 1 ) , (1) and (Θ | X = x 2 ) = st (Θ | X = x 1 ) , if H ( x 2 ) = H ( x 1 ) . (2) Ordinal sufficiency (OS) implies only (1). Robert Zwitser, Gunter Maris Presentation IMPS 2013

The Use of Sum Scores Suffiency Testable Implications Illustrations Future directions Some models and OS of X + OS of X + for θ ∗ ? MHM: no MHM + invariant item ordering + monotone traceline ratio: no Normal Ogive Model: no 2PL model: depends on the discrimination parameters ∗ proofs provided in the paper Robert Zwitser, Gunter Maris Presentation IMPS 2013

The Use of Sum Scores Suffiency Testable Implications Illustrations Future directions Some models and OS of X + OS of X + for θ ∗ ? MHM: no MHM + invariant item ordering + monotone traceline ratio: no Normal Ogive Model: no 2PL model: depends on the discrimination parameters New model: nonparametric Rasch Model (npRM) UD, LI, M and OS. ∗ proofs provided in the paper Robert Zwitser, Gunter Maris Presentation IMPS 2013

The Use of Sum Scores Suffiency Testable Implications Illustrations Future directions Testable Implications of OS (1) Lemma OS of the sum score for a set of items implies OS of the sum score for any subset of items. Lemma If OS of the sum score holds in all subsets of ( p − 1) items, then it also holds for all p items, provided p is even. Robert Zwitser, Gunter Maris Presentation IMPS 2013

The Use of Sum Scores Suffiency Testable Implications Illustrations Future directions Testable Implications of OS (2) Let S be a subset of X . Robert Zwitser, Gunter Maris Presentation IMPS 2013

The Use of Sum Scores Suffiency Testable Implications Illustrations Future directions Testable Implications of OS (2) Let S be a subset of X . The null hypotheses: H ( X ) is ordinal sufficient for θ . Then ∀ s 1 , s 2 : (Θ | s 2 ) > st (Θ | s 1 ) , if H ( s 2 ) > H ( s 1 ) . Robert Zwitser, Gunter Maris Presentation IMPS 2013

The Use of Sum Scores Suffiency Testable Implications Illustrations Future directions Testable Implications of OS (2) Let S be a subset of X . The null hypotheses: H ( X ) is ordinal sufficient for θ . Then ∀ s 1 , s 2 : (Θ | s 2 ) > st (Θ | s 1 ) , if H ( s 2 ) > H ( s 1 ) . These multiple sub-hypotheses can be tested by determining whether ¯ ¯ S S ∀ s 1 , s 2 : ( X + | s 2 ) > st ( X + | s 1 ) , if H ( s 2 ) > H ( s 1 ) . Robert Zwitser, Gunter Maris Presentation IMPS 2013

The Use of Sum Scores Suffiency Testable Implications Illustrations Future directions Example (1) Dutch Entrance Test (in Dutch: Entreetoets ) multiple subtests administered annually to 125,000 grade 5 pupils subtest with 120 math items Robert Zwitser, Gunter Maris Presentation IMPS 2013

The Use of Sum Scores Suffiency Testable Implications Illustrations Future directions Example (2) One-Parameter Logistic Model item discrimination (OPLM): 1 2 2 4 3 3 exp[ a i ( θ − δ i )] P i ( θ ) = · · · · · · 1 + exp[ a i ( θ − δ i )] 6 2 · · · · · · 8 4 · · · · · · 10 5 · · · · · · Robert Zwitser, Gunter Maris Presentation IMPS 2013

The Use of Sum Scores Suffiency Testable Implications Illustrations Future directions Example (3): Results 1.0 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● item discrimination ● ● ● ● ● ● ● ● ● 0.8 ● ● ● 1 2 ● ● ● ● ● ● ● ● 2 4 ● 0.6 ● ● ● S | s ) ● ● ● 3 3 F(X + ● ● ● ● 0.4 ● ● ● ● ● ● ● ● ● ● ● 0.2 ● ● ● Kolmogorov-Smirnov Test: ● ● ● s = {0,0,1} ● ● ● ● D − = 0.0002, p = . 9998 ● ● s = {1,1,0} ● ● ● ● ● ● 0.0 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● 0 20 40 60 80 100 120 S | s X + Robert Zwitser, Gunter Maris Presentation IMPS 2013

The Use of Sum Scores Suffiency Testable Implications Illustrations Future directions Example (3): Results 1.0 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● item discrimination ● ● ● ● ● ● ● ● 0.8 ● ● ● 1 2 ● ● ● ● ● ● ● ● 6 2 ● 0.6 ● ● ● S | s ) ● ● ● ● 10 5 F(X + ● ● ● ● ● 0.4 ● ● ● ● ● ● ● ● ● ● ● 0.2 ● ● ● Kolmogorov-Smirnov Test: ● ● ● ● s = {0,0,1} ● ● ● ● ● D − = 0.0829, p < . 001 ● ● ● s = {1,1,0} ● ● ● ● ● ● 0.0 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● 0 20 40 60 80 100 120 S | s X + Robert Zwitser, Gunter Maris Presentation IMPS 2013