Quasi-Exact Tests for the dichotomous Rasch Model conditional - PowerPoint PPT Presentation

Introduction Rasch model Properties ❨ unidimensionality/homogenous items Quasi-Exact Tests for the dichotomous Rasch Model ❨ conditional independence (local independence) ❨ specific objectivity/sample independence Ingrid Koller, Vienna University Reinhold Hatzinger, WU-Vienna ❨ strictly monotone increasing item characteristic function ❨ sufficient statistics Psychoco 2012, Ingrid Koller & Reinhold Hatzinger 1 Psychoco 2012, Ingrid Koller & Reinhold Hatzinger 2 Introduction Quasi-exact tests Why quasi-exact tests? Quasi-exact tests? Parametric methods need large samples because of with quasi-exact tests it is possible to test the Rasch-model ❨ consistency and unbiasedness of parameter estimates (RM) also with small samples ❨ assumption of asymptotic distribution of test statistics ❨ higher power of test-statistics Sampling binary matrices Small samples in practise description of MCMC method: Kathrin Gruber ❨ large samples often not available (e.g., clinical studies) ❨ complex study designs (e.g., experiments) ❨ smaller costs and less time-consuming development of test-statistics (T) for the dichotomous RM ❨ possibility to test the quality of items also in small samples ❨ Ponocny (1996, 2001) (e.g., stepwise test-construction) ❨ Chen & Small (2005) ❨ Verhelst (2008) ❨ Koller & Hatzinger (in prep.) Psychoco 2012, Ingrid Koller & Reinhold Hatzinger 3 Psychoco 2012, Ingrid Koller & Reinhold Hatzinger 4

Procedure Conditional dependence General procedure for the T-statistics Conditional dependence ❨ A 0 is the observed matrix with the margins r v and c i where r v � P i x vi (person score) and c i � P v x vi (item score) T 11 : large inter-item correlations ❨ Σ rc is the set of all matrices with fixed r and c (sample space) r ij � P S s � 1 r ij T 11 ❼ A ➁ � ◗ ❙ r ij ✏ ➬ r ij ❙ ➬ where S Algorithm ij ❨ sample s � 1 ,...,S matrices A s from Σ rc r ij . . . the inter-item-correlation for item i and item j ➬ ❨ calculate T 0 for the observed matrix A 0 r ij . . . mean of r ij from all simulated matrices ➣ ❨ calulate T 1 ,...,T S for all sampled matrices A 1 ,..., A S ➝ T s ❼ A s ➁ ❈ T 0 ❼ A 0 ➁ ➝ S t s ⑦ S ➛ 1 , ❨ determine your p-value by ◗ where p � t s � ➝ ➝ ↕ else 0 , s � 1 ➣ if r ij in A 0 is large, then the difference r ij ✏ ➬ ➝ ➝ T s ❼ A s ➁ ❈ T 0 ❼ A 0 ➁ r ij is also large S 1 , t s ⑦ S ➛ ◗ p � where t s � ➝ only a few T s show the same or a higher difference than T 0 ➝ ↕ 0 , else s � 1 highly correlated items indicate violation of conditional indepen- dence Psychoco 2012, Ingrid Koller & Reinhold Hatzinger 5 Psychoco 2012, Ingrid Koller & Reinhold Hatzinger 6 Multidimensionality Conditional dependence & Multidimensionality Multidimensionality Conditional dependence T 1 : many equal responses T 11 m : small inter-item-correlations ❨ count the number of ➌ 00 ➑ and ➌ 11 ➑ patterns in items i and j ❨ how many T s have same or a higher value than T 0 ➣ same equation as for T 11 , but modified test: ➝ ➝ ➣ 1 , x vi � x vj T 1 ❼ A ➁ � ◗ ➛ ➝ ➝ T s ❼ A s ➁ ❇ T 0 ❼ A 0 ➁ where δ ij δ ij � ➝ S ➝ 1 , t s ⑦ S ➛ ↕ 0 , x vi ① x vj ◗ v p � where t s � ➝ ➝ ↕ 0 , else s � 1 many equal responses indicate violation of conditional indepen- dence if r ij in A 0 is small, then the difference r ij ✏ ➬ r ij is also small Multidimensionality: only a few T s show the same or a smaller difference than T 0 T 1 m : few equal responses small correlations between items indicate multidimensionality ❨ how many T s have same or a lower value than T 0 few equal responses indicate that the correlation between items is too small, unidimensionality assumption may be violated Psychoco 2012, Ingrid Koller & Reinhold Hatzinger 7 Psychoco 2012, Ingrid Koller & Reinhold Hatzinger 8

Conditional dependence & Multidimensionality Conditional dependence & Multidimensionality Learning Conditional dependence T 1 ℓ : many ➌ 11 ➑ patterns (e.g.,Koller & Hatzinger) T 2 : high dispersion of rawscore r v for a set of items ❨ count only ➌ 11 ➑ patterns as opposed to T 1 ❨ if items are dependent, the variance of r v is large ❨ because of var ❼ z ➁ � var ❼ x ➁ ✔ var ❼ y ➁ ✔ 2 ❻ cov ❼ x,y ➁ ❨ define a set of items I and calculate r ❼ I ➁ ➣ ➝ v ➝ 1 , x vi � x vj � 1 T 1 l ❼ A ➁ � ◗ ➛ ❨ count how many T s ❈ T 0 δ ij where δ ij � ➝ ➝ ↕ else 0 v T 2 ❼ A ➁ � var v ❼ r ❼ I ➁ v ➁ r ❼ I ➁ � ◗ where x vi v ❨ how many T s have same or a higher value than T 0 i ❃ I other possibilities: range, mean absolute deviation, median ab- if person has learned from one item ( x vi � 1 ) then the probability solute deviation. p ❼ x vj � 1 ➁ is increased for a positive reponse to another item j Multidimensionality: T 2 m : low dispersion of rawscore r v for a set of items ❨ count how many T s ❇ T 0 Psychoco 2012, Ingrid Koller & Reinhold Hatzinger 9 Psychoco 2012, Ingrid Koller & Reinhold Hatzinger 10 Multidimensionality Subgroup-invariance Multidimensionality Subgroup-invariance T 10 : based on counts on certain item responses T MU : correlation of rawscore for item subsets ❨ n ij ⑦ n ji is proportional to the ratio of exp ❼ β i ➁⑦ exp ❼ β j ➁ (Koller & Hatzinger) ❨ no parameter differences for focal-group foc and reference- ij ⑦ n ref ij ⑦ n foc group ref : n ref � n foc ji ji ❨ sum of differences for all pairs of items ❨ if two sets of items I are unidimensional, r I v of set I and r J v ❨ counts of T s ❈ T 0 of set J should be positiv correlated ❨ with increasing r I v also r J v should be increasing T 10 ❼ A ➁ � ◗ ❙ n ref ij ❙ ij n foc ✏ n ref ji n foc ji ij ❨ count the number of correlations T s ❇ T 0 ❨ if the parameter differ across groups, the difference should be increasing Note: ❨ external criterion (e.g., gender): uniform DIF T MU ❼ A ➁ � cor ❼ r I v ➁ v ,r J r I v � ◗ x vi � ❨ internal criterion (e.g., rawscore-median): discrimination, guessing, fal- i ❃ I sity ❨ split on specified item: conditional dependence Psychoco 2012, Ingrid Koller & Reinhold Hatzinger 11 Psychoco 2012, Ingrid Koller & Reinhold Hatzinger 12

Subgroup-invariance Subgroup-invariance Subgroup-invariance Subgroup-invariance T DTF : based on item differences on item sumscores T 4 :counts of positive responses in person subgroups (Koller & Hatzinger) ❨ assumption: in one group of persons G one or more items ❨ similar to T 4 , but with the possibility to test DTF (all items are easier/more difficult as expected in the RM in a test shows subgroup-invariance) ❨ count the number of persons who solved these items ❨ calculate the sumscores ( c ) for one item (or a group of items) ❨ easier: counts of T s ❈ T 0 for the reference group c ref and for the focal group c foc i i ❨ more difficult:counts of T s ❇ T 0 ❨ calculate the difference of c between focal and reference group. ❨ easier: counts of T s ❈ T 0 ❨ more difficult:counts of T s ❇ T 0 T 4 ❼ A ➁ � ◗ x vi T DTF ❼ A ➁ � ◗ ❼ c ref ➁ 2 v ❃ G ✏ c foc i i i ❃ I Note: ❨ tests the same assumptions as T 10 Note: ❨ tests the same assumptions as T 10 & T 4 Psychoco 2012, Ingrid Koller & Reinhold Hatzinger 13 Psychoco 2012, Ingrid Koller & Reinhold Hatzinger 14 Unfolding response structure Item discrimination Unfolding response structure - monotonicity Item discrimination T 6 : responses in three person subgroups T 5 : rawscore for persons with x vi � 0 for a certain item ❨ similar to T 4 . ❨ include persons who answer with 0 to a certain item ❨ split the sample in three rawscore groups and count the ❨ sum r v of the remaining items for group x vi � 0 number of positive responses only in the middle group G m ❨ counts of T s ❈ T 0 ❨ easier (reversed U-shape): counts of T s ❈ T 0 ❨ more difficult (U-shape): counts of T s ❇ T 0 T 5 � ❼ A ➁ � ◗ r v v ❙ x vi � 0 ❨ if persons with high ability ( r v � high) fail to solve a certain T 6 ❼ A ➁ � ◗ item, this item may show too low discrimination, falsity, or x vi v ❃ G m indicate multidimensionality Psychoco 2012, Ingrid Koller & Reinhold Hatzinger 15 Psychoco 2012, Ingrid Koller & Reinhold Hatzinger 16