Measurement Measurement Invariance Invariance Ed Merkle, Ed Merkle, Achim Zeileis Achim Zeileis Background Background 1 Background Generalized Measurement Invariance Tests for Proposed Proposed Tests Tests Factor Analysis Illustration Illustration Conclusions Conclusions 2 Proposed Tests Ed Merkle 1 Achim Zeileis 2 3 Illustration 1 University of Missouri 2 Universit¨ at Innsbruck Supported by grant SES-1061334 from the U.S. National 4 Conclusions Science Foundation Measurement Measurement Invariance Invariance Measurement Invariance Example (Age ≤ 16) Ed Merkle, Ed Merkle, Achim Zeileis Achim Zeileis Background Background Proposed Proposed Tests Tests Illustration Illustration Conclusions Conclusions • Measurement invariance: Sets of tests/items consistently assigning scores across diverse groups of individuals. • Notable violations of measurement invariance: • SAT for different ethnic groups (Atkinson, 2001) • Intelligence tests & the Flynn effect (Wicherts et al., 2004)
Measurement Measurement Invariance Invariance Example (Age > 16) Hypotheses Ed Merkle, Ed Merkle, Achim Zeileis Achim Zeileis Background Background Proposed Proposed Tests Tests Illustration Illustration Conclusions Conclusions • Hypothesis of “full” measurement invariance: H 0 : θ i = θ 0 , i = 1 , . . . , n H 1 : Not all the θ i = θ 0 where θ i = ( λ i , 1 , 1 , . . . , ψ i , 1 , 1 , . . . , ϕ i , 1 , 2 ) ⊤ is the full p -dimensional parameter vector for individual i . Measurement Measurement Invariance Invariance Hypotheses Lack of Grouping Ed Merkle, Ed Merkle, Achim Zeileis Achim Zeileis Background Background Proposed • H 0 from the previous slide is difficult to fully assess due to Proposed Tests Tests all the ways by which individuals may differ. 50 LR Illustration Illustration LM Conclusions Conclusions LR and LM statistics (k* = 19) 40 • We typically place people into groups based on a meaningful auxiliary variable, then study measurement 30 invariance across those groups (via Likelihood Ratio tests, Lagrange multiplier tests, Wald tests). 20 • If we did not know the groups in advance, we could 10 conduct a LR or LM test for each possible grouping, then take the maximum. Requires different critical values! (Can 0 be obtained from proposed tests.) 14 15 16 17 Age
Measurement Measurement Invariance Invariance Proposed Tests Proposed Tests Ed Merkle, Ed Merkle, Achim Zeileis Achim Zeileis Background Background Proposed Proposed • The proposed family of tests rely on first derivatives of the Tests Tests model’s log-likelihood function. Illustration Illustration • In contrast to existing tests of measurement invariance, Conclusions Conclusions • We consider individual terms ( scores ) of the gradient. the proposed tests offer the abilities to: These scores tell us how well a particular parameter • Test for measurement invariance when groups are describes a particular individual. ill-defined (e.g., when the grouping variable is continuous). • Test for measurement invariance in any subset of model n � parameters. s (ˆ θ ; x i ) = 0 , where • Interpret the nature of measurement invariance violations. i =1 � θ ; x i ) = ∂ s (ˆ � ∂ θ log L( x i , θ ) θ = � θ Measurement Measurement Invariance Invariance Proposed Tests Aggregating Scores Ed Merkle, Ed Merkle, Achim Zeileis Achim Zeileis Background Background Proposed Proposed Tests Tests • We need a way to aggregate scores across people so that Illustration Illustration we can draw some general conclusions. Conclusions • Under measurement invariance, parameter estimates Conclusions • Order individuals by an auxiliary variable. should roughly describe everyone equally well. So people’s scores should fluctuate around zero. • Define t ∈ (1 / n , n ). The empirical cumulative score process is defined by: • If measurement invariance is violated, the scores should ⌊ nt ⌋ � 1 stray from zero. B (ˆ s (ˆ θ ; t ) = √ n θ ; x i ) . i =1 where ⌊ nt ⌋ is the integer part of nt .
Measurement Measurement Invariance Invariance Tests Simulation Ed Merkle, Ed Merkle, Achim Zeileis Achim Zeileis • Under the hypothesis of measurement invariance, a Background Background functional central limit theorem holds: Proposed Proposed Tests Tests Illustration Illustration θ ; · ) d I ( � θ ) − 1 / 2 B ( � → B 0 ( · ) , • Simulation: What is the power of the proposed tests? Conclusions Conclusions where I ( � θ ) is the observed information matrix and B 0 ( · ) is • Two-factor model, with three indicators each. a p -dimensional Brownian bridge. • Measurement invariance violation in three factor loading parameters, with magnitude from 0–4 standard errors. • Testing procedure: Compute an aggregated statistic of the • Sample size in { 100 , 200 , 500 } . • Model parameters tested in { 3 , 19 } . empirical score process and compare with corresponding • Three test statistics. quantile of aggregated Brownian motion. • Test statistics: Special cases include double maximum (DM), Cram´ er-von Mises (CvM), maximum of LM statistics. Measurement Measurement Invariance Invariance Simulation Example Ed Merkle, Ed Merkle, Achim Zeileis Achim Zeileis CvM Background max LM Background DM Proposed Proposed 0 1 2 3 4 Tests Tests n = 500 n = 500 k* = 3 k* = 19 Illustration Illustration • Example: Studying stereotype threat via factor analysis 0.8 Conclusions Conclusions (Wicherts et al., 2005) 0.6 0.4 • Stereotype threat: Knowledge of stereotypes about one’s 0.2 social group might cause one to fulfill the stereotypes. n = 200 n = 200 k* = 3 k* = 19 • Wicherts et al. study: 295 students were administered 0.8 three intelligence tests. Stereotypes were primed for half of Power 0.6 the students. 0.4 0.2 • Groups defined by: Ethnicity (majority/minority) and n = 100 n = 100 whether or not stereotypes were primed. k* = 3 k* = 19 0.8 0.6 0.4 0.2 0 1 2 3 4 Violation Magnitude
Measurement Measurement Invariance Invariance Model Model Ed Merkle, Ed Merkle, Achim Zeileis Achim Zeileis Background Background Proposed Proposed Tests Tests Illustration Illustration • We utilize a model employed by Wicherts et al., where Conclusions Conclusions • To study the data, Wicherts et al. employed a series of four model parameters are specific to the “minority, four-group, one-factor models. stereotype prime” group. • General finding: Minorities with stereotype primes have • Test for measurement invariance in these parameters wrt different measurement parameters than other groups. the student GPA variable (either all four together or • Current example: Is measurement further impacted by individually). academic performance (as measured by student GPA)? • Violations of measurement invariance imply that stereotype threat is more problematic for students of low or high GPA. Measurement Measurement Invariance Invariance Model Results for Single Parameters Ed Merkle, Ed Merkle, Achim Zeileis Achim Zeileis Background Background Proposed Proposed 2 2 Tests Tests ψ num 1 1 Illustration Illustration λ n 0 η 0 μ num Conclusions Conclusions Numerical −1 −1 λ num −2 −2 4.5 5.5 6.5 7.5 4.5 5.5 6.5 7.5 1 Abstract Intelligence GPA GPA η Verbal 2 2 1 1 ψ n µ n 0 0 −1 −1 −2 −2 4.5 5.5 6.5 7.5 4.5 5.5 6.5 7.5 GPA GPA
Measurement Measurement Invariance Invariance Aggregated Results Conclusions Ed Merkle, Ed Merkle, Achim Zeileis Achim Zeileis Background Aggregated Process, Double Max Aggregated Process, CvM Background Empirical fluctuation process Empirical fluctuation process Proposed Proposed 6 • Measurement invariance tests utilizing stochastic processes 2.0 Tests Tests 5 have important advantages over existing tests: 4 Illustration Illustration 1.0 3 • Isolating specific parameters that violate measurement Conclusions Conclusions 2 invariance, allowing the researcher to define specific types 1 0.0 0 of measurement invariance “post hoc” instead of “a 4.5 5.5 6.5 7.5 4.5 5.5 6.5 7.5 priori”. GPA GPA • Isolating groups of individuals whose parameter values differ. Aggregated Process, max LM • Studying the impact of continuous variables on model Empirical fluctuation process 30 estimates, without “ruining” the rest of the model. 20 • Power is reasonable, with specific tests being better in specific circumstances. 10 5 5.5 6.0 6.5 7.0 GPA Measurement Measurement Invariance Invariance Software Current Work Ed Merkle, Ed Merkle, Achim Zeileis Achim Zeileis Background Background Proposed Proposed Tests Tests • Continued test implementation via strucchange and • To carry out the tests, we utilize Illustration Illustration lavaan (and possibly OpenMx ). • lavaan for model estimation. Conclusions Conclusions • estfun() for score extraction, which is currently a • Detailed examination of test properties. combination of our own code and lavaan code. • strucchange for carrying out the proposed tests with the scores. • Extension to related psychometric issues. • Required input: Fitted model, function for score extraction, and information matrix (optional). • gefp() constructs the process. • Working paper: • sctest() and plot() calculate and visualize test http://econpapers.repec.org/RePEc:inn:wpaper: statistics. 2011-09
Measurement Invariance Ed Merkle, Achim Zeileis Background Proposed Tests Illustration Conclusions • Questions?
Recommend
More recommend
