Knowledge structures (Doignon & Falmagne, 1985, 1999) Goals - PowerPoint PPT Presentation

Probabilistic knowledge structures The pks package Parameter estimation Outlook Knowledge structures (Doignon & Falmagne, 1985, 1999) Goals Parameter estimation in probabilistic knowledge • Characterizing the strengths and weaknesses in all parts of a structures with the pks package knowledge domain • Precise, non-numerical characterization of the state of knowledge that is computationally tractable • Building upon results from discrete mathematics and exploiting Florian Wickelmaier and J¨ urgen Heller the power of current computers • Adaptive knowledge assessment • Efficiently identifying the current state of knowledge based on asking a minimal number of questions • Adapting to the already given responses as experienced Psychoco 2012, Innsbruck, February 9 teachers do in an oral examination • Personalization in technology-enhanced learning • Automatically select content that a person is ready to learn 2 Probabilistic knowledge structures The pks package Parameter estimation Outlook Probabilistic knowledge structures The pks package Parameter estimation Outlook A subdomain of physics: Conservation of matter (1) A subdomain of physics: Conservation of matter (2) (Taagepera et al., 1997) (Taagepera et al., 1997) d) After 3 metal nuts and 3 metal bolts are joined together: a) When ice melts and produces water: (i) The total amount of metal is the same. (i) The water weighs more than the ice. (ii) There is less metal than before. (ii) The ice weighs more than the water. (iii) There is more metal than before. (iii) The water and ice weigh the same. (iv) The amount of metal cannot be determined. (iv) The weight depends on the temperature. e) Photosynthesis can be described as: b) After the nail rusts, its mass: (i) is greater than before. chlorophyll WATER + CARBON DIOXIDE − − − − − − → GLUCOSE (ii) is less than before. sunlight (iii) is the same as before. Which of the following statements about this reaction is NOT (iv) cannot be predicted. true? c) When 10 grams of iron and 10 grams of oxygen combine, the (i) As more water and more carbon dioxide react, more glucose is total amount of material after iron oxide (rust) is formed must produced. weigh: (ii) The same amount of glucose is produced no matter how much (i) 10 grams. water and carbon dioxide is available. (ii) 19 grams. (iii) Chlorophyll and sunlight are needed for the reaction. (iii) 20 grams. (iv) The same atoms make up the GLUCOSE molecule as were (iv) 21 grams. present in WATER and CARBON DIOXIDE. 3 4

Probabilistic knowledge structures The pks package Parameter estimation Outlook Probabilistic knowledge structures The pks package Parameter estimation Outlook 11111 ● Response patterns Deterministic theory 01111 ● 10111 ● 11011 ● (Taagepera et al., 1997) 11101 ● 11110 ● 00111 ● 01011 ● Definitions 01101 ● 01110 ● 10011 ● • A knowledge domain is identified with a set Q of 10101 ● 10110 ● Students from grades (dichotomous) items. 11001 ● 11010 four through twelve ● • The knowledge state of a person is identified with the subset 11100 ● 00011 N = 1620 ● K ⊆ Q of problems in the domain Q the person is capable of 00101 ● 00110 ● solving. 01001 ● 01010 ● • A knowledge structure on the domain Q is a collection K of 01100 ● 10001 ● subsets of Q that contains at least the empty set ∅ and the 10010 ● 10100 ● set Q . 11000 ● 00001 ● 00010 ● • The subsets K ∈ K are the knowledge states. 00100 ● 01000 ● 10000 ● 00000 ● 0 50 100 150 200 250 Frequency 5 6 Probabilistic knowledge structures The pks package Parameter estimation Outlook Probabilistic knowledge structures The pks package Parameter estimation Outlook Conservation of matter: Knowledge structure Probabilistic knowledge structures (Taagepera et al., 1997) Rationale • If there are response errors then knowledge states K ⊆ Q and Q response patterns R ⊆ Q have to be dissociated. Definition abce acd • A probabilistic knowledge structure is defined by specifying acde • a knowledge structure K on a knowledge domain Q (i. e., a bce ac collection K ⊆ 2 Q with ∅ , Q ∈ K ) ad ade • a marginal distribution P K ( K ) on the knowledge states K ∈ K bc • the conditional probabilities P ( R | K ) to observe response a pattern R given knowledge state K de c The probability of the response pattern R ∈ R = 2 Q is predicted by d e � P R ( R ) = P ( R | K ) P K ( K ) ∅ K ∈K 7 8

Probabilistic knowledge structures The pks package Parameter estimation Outlook Probabilistic knowledge structures The pks package Parameter estimation Outlook The basic local independence model (BLIM) The pks package (Doignon & Falmagne, 1999) Assumption: Local stochastic independence • Provides functionality for parameter estimation in probabilistic • Given the knowledge state K of a person knowledge structures. • the responses are stochastically independent over problems • Main functions • the response to each problem q only depends on the Fitting and testing basic local probabilities blim β q of a careless error independence models (BLIMs) η q of a lucky guess Extractor functions print, logLik • The probability of the response pattern R given the knowledge plot, residuals state K reads simulate generate response patterns from a given BLIM � � � � P ( R | K ) = (1 − β q ) (1 − η q ) . β q η q conversion functions as.pattern,as.binmat q ∈ K \ R q ∈ K ∩ R q ∈ R \ K q ∈ Q \ ( R ∪ K ) 9 10 Probabilistic knowledge structures The pks package Parameter estimation Outlook Probabilistic knowledge structures The pks package Parameter estimation Outlook Maximum likelihood estimation Example: Maximum likelihood estimation EM algorithm 0 . 2 . 4 β a Q • Formulate the likelihood as if we have available the absolute β b frequencies M RK of subjects who are in state K and produce β c abce pattern R (complete data) instead of the absolute frequencies acd β d N R of the response patterns R ∈ R (incomplete data). acde β e bce ac Expectation Maximization 0 . 2 . 4 ad ade η a Estimate ˆ β ( t +1) , ˆ η ( t +1) , ˆ π ( t +1) bc Compute a η b based on m RK = E ( M RK ) de E ( M RK ) = η c c d N R · P ( K | R , ˆ β ( t ) , ˆ η ( t ) , ˆ π ( t ) ) η d e η e ∅ 11 12

Probabilistic knowledge structures The pks package Parameter estimation Outlook Probabilistic knowledge structures The pks package Parameter estimation Outlook Example: Maximum likelihood estimation Maximum likelihood estimation Problems blim(matter97$K, matter97$N.R, method="ML") • ‘Good fit’ (w.r.t. likelihood ratio statistic) not sufficient for empirical validity of knowledge structure Number of iterations: 9474 Goodness of fit (2 log likelihood ratio): • Fit may be obtained by inflating careless error rates β q and G2(7) = 13.763, p = 0.055553 lucky guess rates η q • What we want: Good fit with small values of β q and η q Minimum discrepancy distribution (mean = 0.2858) • How to apply constraints on the error probabilities that are 0 1 1157 463 motivated by the knowledge structure? (instead of brute-force constraints, Stefanutti & Robusto, Mean number of errors (total = 1.02435) 2009) careless error lucky guess • How much of the fit is due to inflating the error probabilities 0.3697625 0.6545893 in ML estimation? 13 14 Probabilistic knowledge structures The pks package Parameter estimation Outlook Probabilistic knowledge structures The pks package Parameter estimation Outlook Minimum discrepancy method Minimum discrepancy method Rationale Rationale • For a given response pattern R , consider the minimum of the • For a response pattern R and a knowledge state K consider symmetric distances between R and all the knowledge states K ∈ K the distance d ( R , K ) = min K ∈K d ( R , K ) . d ( R , K ) = | ( R \ K ) ∪ ( K \ R ) | , which is based on the symmetric set-difference. • The basic idea is that any response pattern is assumed to be generated by a close knowledge state • It is the number of items that are elements of either, but not both sets R and K (number of response errors). • leads to explicit (i. e., non-iterative) estimators of the error probabilities • Example • minimizes the number of response errors and thus counteracts d (10001 , 10100) = 2 an inflation of careless error and lucky guess probabilities • A previously suggested implementation of this idea by Schrepp (1999, 2001) does not allow for item specific estimates. 15 16