17 th International Meeting of the Psychometric Society Hong Kong 2011 How I Ho Ite tem m Resp sponse onse Th Theor ory y can sol olve ve ta prob oble lems ms of of i ipsativ ative e da data Doctoral student: Anna Brown Advisor: Prof. Alber berto to Maydeu deu-Oliva livares es
Mot otiv ivation ation fo for th this is wo work • Practical problem that desperately needed a solution – Old – Very widespread (mainly workplace selection and assessment tools, millions of administrations per year) – Thousands of pages in journals over years have been devoted to the problem – Psychometrics with all its sophisticated methodologies had failed to provide a solution 2
Curse of ipsative data Fo Forced-cho choic ice e respo ponse nse f for ormat mat 3
4
5
6
7
8
Fo Forced-cho choic ice e fo format • Multidimensional Forced-Choice (MFC) format – Rank-order two or more items from different dimensions A. I manag age e to relax x easily B. I am ca caref eful ul over er detail ail C. I enjoy joy workin king g with h othe hers rs 9
Ad Advantages ntages of of c com ompar arative ative fo forma mats ts – Designed to reduce response biases • Direct comparison overcomes problems with interpretation of the rating scale • Impossible to endorse all items thus: – eliminates uniform biases / response sets (acquiescence, extreme/central tendency responding) – Reduces halo/horn effects – May reduce effects of socially desirable responding 10
Cl Classic ssical al scori oring ng of of F FC f C for ormats ts • Inverted rank orders of items are added to their respective scales Most Least Classical score A. I mana nage ge to relax ax easi sily ly 1 B. I am carefu eful l over detai ail 2 C. I enjoy oy worki rking ng with th others ers 0 – The same number of points in each block is allocated for any individual – The total score on the test is constant for each individual ( ipsative data) 11
Prob oble lems ms of of i ipsati ative ve da data ta 1. Scores are relative – Impossible to get all high/low scores – Intra-individual comparisons are problematic 2. Construct validity is distorted – Variance of the total test score is zero 1 r – Negative average scale inter-correlation d 1 3. Criterion-related validity is distorted – Correlations with an external criterion must sum to zero – Compensatory correlations 4. Reliability estimates are distorted – Basic assumptions are violated ( Cronbach’s alpha and other coefficients) 12
Inade dequate quate scor oring ing • Classical scoring methodology is inadequate for forced-choice items – Rankings are treated as ratings (i.e. relative scores are treated as absolute) – Items within each block are NOT assessed independently • Need to radically depart from classical scoring schemes – Modelling the psychological process of responding to forced-choice items is the key to making sense of comparative data • Suitable psychological models for such data have existed for a long time, and they are well known 13
Modelling decision process behind responding to forced- choice items Th Thurston rstonia ian IRT T mo mode del 14
Ps Psychol ychologi ogical cal valu lue • Louis Thurstone (1927-1929) introduced the notion of a psychological value or utility – Describes “the affect that the object calls forth”; – Varies across individuals for the same object, and across objects within the same individual; – Can be placed on a psychological continuum; – Assumed normally distributed across individuals. • The notion of utility maximisation – when confronted with choosing between two items, respondents will choose the item with the highest psychological value (utility). 15
La Law of of c com omparati rative ve ju judg dgeme ment nt ( (19 1927) 7) • A respondent prefers item i to item k , if her or his utility t i is larger than t k 1, if t t i k y y i k , 0, if t t l l i k • The difference of two utilities is normally * y t t l i k distributed • Binary outcome of comparison y l linked to y l * through a threshold process * 1, if y 0 l y y i k , * l l 0, if y 0 l 16
Bin inary y cod odin ing of of r ranki king ng da data ta • Any ordering or ranking of n choice alternatives requires n(n-1)/2 separate comparison judgements: – Rank-ordering of 2 items elicits 1 comparison: {A,B} – Rank-ordering of 3 items elicits 3 comparisons: {A,B} {A,C} {B,C} – Rank-ordering of 4 items elicits 6 comparisons: {A,B} {A,C} {A,D} {B,C} {B,D} {C,D} – Etc. – “Most” – “least” format with 4 or more alternatives is partial ranking • Any comparison { i, k } is coded as 1 if i preferred to k , and 0 otherwise • Ordering {B, A, C} can be equivalently presented as 3 outcomes {A, B} {A, C} {B, C} 0 1 1 17
Thursto rstonian nian factor or models els 2 • Maydeu-Olivares (1999); Maydeu- 1 0 1 Olivares & Böckenholt (2005) t * y 1 1,2 1 -1 2 1 0 2 • Outcomes of comparisons are 1 * 1 y t determined by the difference in 1,3 2 -1 2 2 0 3 utilities (no error terms) 1 -1 * y t 2,3 3 • Second-order factors (traits) can 3 2 0 4 be modelled 21 4 1 * y t 4,5 4 • Identification constraints are -1 2 1 0 5 needed 1 * 2 y 5 t 4,6 5 -1 – Fixing uniqueness of one utility per 2 31 0 6 1 block -1 * y t 5,6 6 – Factor variances are fixed to 1 2 0 7 7 • Special identification cases 1 * y t 6 32 7,8 7 – Pairs of items -1 2 1 0 8 8 1 * 3 y 7,9 t 8 -1 2 0 9 1 9 * -1 18 y 8,9 t 9
IRT T reparame ameter terization ization • Thurstonian second-order models cannot be used directly in person- centric applications – we are interested in persons’ traits (second-order factors), not the utilities (first-order factors) – but the latent traits cannot be estimated • Re-parameterization as an IRT model (first-order) • Utilities of items i and k are functions of underlying factors (traits) a and b : t i = i + i a + i t k = k + k b + k • Latent difference of utilities y l * = t i -t k is a function of the traits: y l = t i - t k =( i - k )+( i a - k b )+( i - k ) = = - k +( i a - k b )+( i - k ) 19
Item em response onse func nctio tion • The IRF for the binary outcome variable y l , which is the result of comparison between items i and k measuring traits a and b , is Pr y 1 , Φ l i a k b l a b 2 2 i k • In intercept / slope form: Pr y 1 Φ l a b l i a k b 1.0 • Special case of same-trait P(y=1) 0.5 (one-dimensional) comparisons 3 0.0 1.5 3 0 2 trait 1 Pr y 1 Φ ( 1 -1.5 l l i k 0 trait 2 -1 -3 -2 -3 20
Thursto rstonian nian IRT T model el 2 2 1 2 • Outcomes of comparisons are * y 1,2 indicators of common factors 2 2 1 1 2 1 3 2 1 1 * (traits) 2 1 y 1,3 2 2 3 3 2 2 4 2 3 • Special features for blocks of 3 or * y 2,3 more items 7 3 4 2 2 21 4 5 2 – Factor loadings are structured * y 4,5 1 2 2 – Uniquenesses are structured 4 6 5 2 4 2 * 2 y – Structured local dependencies 4,6 5 5 31 6 2 6 2 2 8 5 6 • Identification constraints are the * y 5,6 same as for the second-order 32 2 2 7 8 model * y 7,8 8 7 1 2 2 7 9 6 2 7 * 3 2 y 9 8 7,9 2 9 2 2 9 8 9 21 * y 8,9
Estim timation ation and sc d scor oring ing • Estimated with general-purpose SEM software M plus (Muthén & Muthén, 1998-2010) • Limited information methods are the only option for most applications – When partial ranking format is used, Bayesian MI are recommended • Respondents' traits levels are estimated by the MAP method – Computationally efficient and unaffected by the number of latent traits 22
Ite tem in info formation tion fu functio tion • Direction of information is considered • Information in direction of trait a for one binary outcome 2 2 corr i k a b l i a k b a , l a b P , 1 P , l a b l a b – Smaller when traits are positively correlated 0.6 Information in direction of 0.5 • One-dimensional case 0.4 trait 1 0.3 3 0.2 1.5 2 2 0.1 ( ) 0 i k l i k 0.0 trait 1 -1.5 3 l 2 P 1 P 1 0 -3 -1 l l trait 2 -2 -3 23
Te Test t in info formation tion and re d reli liabil bility ity • Test information in direction of trait a a a l l • When using posterior latent trait estimator 2 ln a a a a P a 2 a 1 • Standard error for trait a SE a a P • Empirical reliability (for estimated scores in a sample) 1 N 1 2 2 2 P error error N a j 1 2 j P P 24
Empirical applications Ap Appli lication ation – CC CCSQ 25
Recommend
More recommend
