Shrinkage estimation of the three-parameter logistic model Michela - PowerPoint PPT Presentation

Shrinkage estimation of the three-parameter logistic model Michela Battauz (joint with Ruggero Bellio) Department of Economics and Statistics University of Udine (Italy) Psychoco 2020 Michela Battauz Shrinkage estimation of the 3PL model Psychoco 2020 1 / 20

The three-parameter logistic (3PL) model Used for modelling the responses of a proficiency test with binary response items, when the probability of guessing is not zero. Probability of a correct response p ij = Pr ( X ij = 1 ∣ θ i ; a j , b j , c j ) to item j exp { a j ( θ i − b j )} p ij = c j + ( 1 − c j ) 1 + exp { a j ( θ i − b j )} , θ i ability of person i 1.0 Item parameters: a j discrimination parameter 0.8 b j difficulty parameter 0.6 c j guessing parameter Prob 0.4 0.2 0.0 −6 −4 −2 0 2 4 6 θ Michela Battauz Shrinkage estimation of the 3PL model Psychoco 2020 2 / 20

Maximum likelihood estimation A convenient parameterization of the model, suitable for estimation is exp ( β 1 j + β 2 j θ i ) p ij = c j + ( 1 − c j ) 1 + exp ( β 1 j + β 2 j θ i ) , with exp ( β 3 j ) c j = 1 + exp ( β 3 j ) . Marginal Maximum Likelihood Estimation ( MLE ) (Bock and Aitkin, 1981) usally adopted, where θ i ∼ N ( 0 , 1 ) and the log-likelihood function is n J log ∫ R ∏ p x ij ij ( 1 − p ij ) 1 − x ij φ ( θ i ) d θ i , ℓ ( β ) = ∑ i = 1 j = 1 Michela Battauz Shrinkage estimation of the 3PL model Psychoco 2020 3 / 20

The 3PL model in practice Broadly used in applications Included in all major IRT software, including R packages ltm , mirt and TAM . Guessing parameters weakly identifiable (Patz and Junker, 1999), MLE has convergence problems. The estimates of the guessing parameters tend to have a negative bias . n = 200 n = 500 n = 1000 n = 5000 200 150 150 200 150 150 100 100 100 100 50 50 50 50 0 0 0 0 −25 −15 −5 −12 −8 −4 0 −8 −6 −4 −2 0 −4 −3 −2 −1 ^ ^ ^ ^ β β β β 3 3 3 3 Michela Battauz Shrinkage estimation of the 3PL model Psychoco 2020 4 / 20

Our proposal: shrinkage estimation of the 3PL model Two main approaches: Penalized maximum likelihood estimation Model-based shrinkage estimation Michela Battauz Shrinkage estimation of the 3PL model Psychoco 2020 5 / 20

Penalty on the guessing parameters Penalized log-likelihood: ℓ p ( β ) = ℓ ( β ) + J ( β 3 ) Penalty proportional to the log p.d.f. of the normal distribution J J ( β 3 ) = − 1 ( β 3 j − µ ) 2 , ∑ 2 σ 2 j = 1 (implemented also in the mirt package) Ridge-type penalty J ( β 3 ) = − λ ∑ ( β 3 j − β 3 k ) 2 , j < k The two penalties are equivalent when µ = J − 1 ∑ j β 3 j , but the ridge-type penalty has only one tuning parameter. Empirical results were very similar, so we chose the ridge-type penalty . Michela Battauz Shrinkage estimation of the 3PL model Psychoco 2020 6 / 20

Model-based shrinkage estimation Application of the bias-reduction method ( BR ) proposed by Firth (1993) for a general parametric model, with estimating equation for β S ( β ) = ∇ ℓ ( β ) − I ( β ) − 1 b ( β ) , where I ( β ) is the expected Fisher information, and b ( β ) is the leading term of the asymptotic bias of the MLE. The estimator defined by solving S ( β ) = 0 has reduced finite sample bias, though it is asymptotically equivalent to the MLE. In many models for discrete data, a useful side effect of bias reduction is shrinkage of parameter estimates (Kosmidis, 2014). Michela Battauz Shrinkage estimation of the 3PL model Psychoco 2020 7 / 20

Implementation Both approaches were implemented the R package S3PL github.com/micbtz/S3PL . Integral approximated using Gaussian quadrature . Penalized likelihood: package Rcpp to speed up computational time; tuning parameter λ selected using cross-validation ; cross-validation error : the negative log-likelihood Bias reduction: package TMB for automatic differentiation and C++ implementation; Monte Carlo evaluation of required expected values; Parallel computing; Michela Battauz Shrinkage estimation of the 3PL model Psychoco 2020 8 / 20

An illustrative example Achievement data collected on students attending the third year of high school in Italy, tested in Mathematics n = 3843 students, J = 14 items Estimated correlation matrix of ˆ β MLE ß11 ß21 ß31 ß12 ß22 ß32 ß13 ß23 ß33 ß14 ß24 ß34 ß11 ß21 ß31 ß12 ß22 ß32 ß13 ß23 ß33 ß14 ß24 ß34 Michela Battauz Shrinkage estimation of the 3PL model Psychoco 2020 9 / 20

Ridge-type penalization 3316.4 0.30 3316.2 0.20 CV error ^ j c 3316.0 0.10 3315.8 0.00 0.0 0.5 1.0 1.5 2.0 0.0 0.5 1.0 1.5 2.0 λ λ Michela Battauz Shrinkage estimation of the 3PL model Psychoco 2020 10 / 20

Comparison of estimates of the guessing parameters 0.5 0.5 0.4 0.4 ● ● 0.3 0.3 ● ● ● ● ● ● ridge BR ● ● ● ● ● ● ● ● 0.2 ● 0.2 ● ● ● ● ● ● ● ● ● 0.1 0.1 ● ● 0.0 0.0 0.0 0.1 0.2 0.3 0.4 0.5 0.0 0.1 0.2 0.3 0.4 0.5 MLE MLE Ridge-type penalization yields a larger shrinkage of the parameters than BR. Michela Battauz Shrinkage estimation of the 3PL model Psychoco 2020 11 / 20

A simulation study True item parameters of 30 items taken from TIMSS 2015, Fourth Grade, Mathematics True guessing parameters: TIMSS (very variable) constant c j = 0 . 2 ∀ j Sample size n = 200 , 500 , 1000 500 replications for each setting Michela Battauz Shrinkage estimation of the 3PL model Psychoco 2020 12 / 20

Results of the simulation study n = 200 constant guessing parameters 7 MLE ridge 6 BR 5 4 3 2 1 0 RMSE ( β 1 ) B ( β 1 ) RMSE ( β 2 ) B ( β 2 ) RMSE ( β 3 ) B ( β 3 ) Michela Battauz Shrinkage estimation of the 3PL model Psychoco 2020 13 / 20

Results of the simulation study n = 500 constant guessing parameters 2.0 MLE ridge BR 1.5 1.0 0.5 0.0 RMSE ( β 1 ) B ( β 1 ) RMSE ( β 2 ) B ( β 2 ) RMSE ( β 3 ) B ( β 3 ) Michela Battauz Shrinkage estimation of the 3PL model Psychoco 2020 14 / 20

Results of the simulation study n = 1000 constant guessing parameters 1.5 MLE ridge BR 1.0 0.5 0.0 RMSE ( β 1 ) B ( β 1 ) RMSE ( β 2 ) B ( β 2 ) RMSE ( β 3 ) B ( β 3 ) Michela Battauz Shrinkage estimation of the 3PL model Psychoco 2020 15 / 20

Results of the simulation study n = 200 variable guessing parameters MLE 5 ridge BR 4 3 2 1 0 RMSE ( β 1 ) B ( β 1 ) RMSE ( β 2 ) B ( β 2 ) RMSE ( β 3 ) B ( β 3 ) Michela Battauz Shrinkage estimation of the 3PL model Psychoco 2020 16 / 20

Results of the simulation study n = 500 variable guessing parameters 2.0 MLE ridge BR 1.5 1.0 0.5 0.0 RMSE ( β 1 ) B ( β 1 ) RMSE ( β 2 ) B ( β 2 ) RMSE ( β 3 ) B ( β 3 ) Michela Battauz Shrinkage estimation of the 3PL model Psychoco 2020 17 / 20

Results of the simulation study n = 1000 variable guessing parameters 1.5 MLE ridge BR 1.0 0.5 0.0 RMSE ( β 1 ) B ( β 1 ) RMSE ( β 2 ) B ( β 2 ) RMSE ( β 3 ) B ( β 3 ) Michela Battauz Shrinkage estimation of the 3PL model Psychoco 2020 18 / 20

Conclusions MLE seems inaccurate even for large sample sizes. The BR method performs well for small sample sizes. For larger samples, ridge-type penalty performs better, especially when the true guessing parameters are constant. Michela Battauz Shrinkage estimation of the 3PL model Psychoco 2020 19 / 20

References Bock, R. D., Aitkin, M. (1981). Marginal maximum likelihood estimation of item parameters: Application of an EM algorithm. Psychometrika, 46, 443–459. Firth, D. (1993). Bias reduction of maximum likelihood estimates. Biometrika, 80, 27-38. Kosmidis, I. (2014). Bias in parametric estimation: reduction and useful side-effects. Wires Comp. Statist., 6, 185-196. Patz, R. J., Junker, B. W. (1999). Applications and extensions of MCMC in IRT: Multiple item types, missing data, and rated responses. J. Educ. Behav. Stat., 24, 342–366. Thank you for your attention! Michela Battauz Shrinkage estimation of the 3PL model Psychoco 2020 20 / 20