Ca Can mo model els le learned fr from a a da datase set re - PDF document

4/20/18 Ca Can mo model els le learned fr from a a da datase set re refl flect ac acquisi3o 3on of proce ocedural kno knowl wledg dge? An An ex experiment wi with au automa3c me measu sureme ment of of on online e re review qua quality Mar$na Megasari, Pandu Wicaksono, Chiao Yun Li, Clément Chaussade, Shibo Cheng, Nicolas Labroche, Patrick Marcel , Verónika Peralta DOLAP 2018 Co Contribu)ons q (Yet another) model of reviews helpfulness A first assessment of the skill of wri9ng helpful reviews q Showing that skill assessment makes sense even for models learned q automa9cally 2 1

4/20/18 Pr Principle … Review 1 Review n Reviewer Helpfulness Helpfulness 3 Principle Pr … Review 1 Review n Probability the skill is Reviewer Helpfulness acquired Helpfulness Skill acquisi6on model 4 2

4/20/18 Skill acquisition model Pr Principle Probability the skill is acquired Helpfulness model Helpfulness model Feature extrac9on … Review 1 Review n Probability the skill is Reviewer Helpfulness acquired Helpfulness Skill acquisi9on model 5 Tracin cing h help lpfuln lness f for a or a r revie iewer 1 0,9 0,8 0,7 Helpfulness score 0,6 0,5 0,4 0,3 0,2 0,1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 Posi:on in the sequence of reviews helpfulness (votes) 6 3

4/20/18 Tracin cing s skill of ill of t the r revie iewer r 1 0,9 0,8 0,7 Helpfulness score 0,6 0,5 0,4 0,3 0,2 0,1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 Posi:on in the sequence of reviews helpfulness (votes) KT helpfulness 7 Tr Tracing helpfulness of the model learned 1 0,9 0,8 0,7 Helpfulness score 0,6 0,5 0,4 0,3 0,2 0,1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 Position in the sequence of reviews helpfulness (votes) model of helpfulness KT helpfulness 8 4

4/20/18 Tr Tracing skill of the model learned 1 0,9 0,8 0,7 Helpfulness score 0,6 0,5 0,4 0,3 0,2 0,1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 Posi:on in the sequence of reviews helpfulness (votes) model of helpfulness KT helpfulness KT model 9 Wh What do we need? Skill acquisi+on model q Bayesian Knowledge Tracing § Data q § Amazon.com book reviews Model q Linear combina:on of features that par:cipate in helpfulness § 10 5

4/20/18 Sk Skill ill ac acquisi*on on: Ba Bayes esia ian Kn Knowledge Tr Tracing (K (KT) User-centric paradigm for evalua5ng procedural knowledge [Corbe' & Anderson, UMAI 1995] Skill S OUTPUT: P(Ln) Probability that skill S is mastered aCer exercice n 11 Hypo Hypotheses heses behi behind nd KT It targets procedural knowledge q Knowledge about how to do something § § Applica5on of procedural knowledge may not be easily explained Different from declara5ve knowledge, that is o?en verbalized § Problem resolu5on is binary q § Pass/fail scheme No forge:ng q 4 parameters usually set empirically q 12 6

4/20/18 The The 4 pa parameters of of KT KT q P(L0): initial knowledge § Probability the skill is already mastered before the first problem q P(T): transition from not mastered to mastered § Probability the skill will be learned at each new opportunity q P(g): Guess § Probability the learner will guess correctly while the skill is not mastered q P(s): Slip § Probability the learner will make a mistake while the skill is mastered 13 Definition De Probability the skill is mastered at step n q P(Ln|Xn =xn) = P(Ln−1|Xn =xn) + (1 − P(Ln−1|Xn =xn))P(T) § § Intui3on : probability the skill is learned at step n-1 or not learned at step n-1 but learned at this step n q With Xn = 1 means problem n resolved sucessfully, Xn=0 means not resolved § § P(Ln−1|Xn =1) = P(Ln−1)(1 − P(s)) / (P (Ln−1)(1 − P(s)) + (1 − P (Ln−1))P(g)) intui3on: skill has been learned and used correctly / all cases of correct resolu3on q P(Ln−1|Xn =0) = P(Ln−1)P(s) / (P (Ln−1)P(s) + (1 − P (Ln−1))(1 − P(g)) ) § 14 7

4/20/18 KT KT extension ions we im imple lemented to o fi fit t ou our conte co text q Non-binary problem resolution § KT with partial credits [Wang & Heffernan, AIED 2013] q Parameter learning avoiding local minimum, degenerate parameters and computational costs § Estimating the most likely opportunity at which each individual learned the skill [Hawkins & al., ITS 2014] q Github link § https://github.com/Cubiccl/Continuous-Knowledge-Tracing/releases/tag/1.0 15 Data: Da a: Amaz mazon book re reviews [H [He & & Mc McAuley, , WWW 2016] In our context q the skill is that of wri.ng helpful reviews § § each wri5en review is treated like an opportunity to exercise the skill Actual helpfulness is the ra.o of helpful § votes received by the review Preprocessing details in the paper q 16 8

4/20/18 Fe Features & metrics for the model of help lpfuln lness q 16 features grouped in 3 categories § Conformity q Rating, polarity, deviation to average rating § Understandability q Spelling error ratio, 5 classical readability measures § Extensiveness q Text and summary length q Consistent with other models in the literature [Korfiatis & al., ECRA 2012] § More sophisticated models exist, but our point was to test a “simple” one 17 The The model del Linear combina,on of feature scores q Learned with linear regression, perceptron, q SVM Regression was the best compromise between 2me § and effec2veness § Feature selec2on had no significant impact 18 9

4/20/18 Te Tests q Implementation § Java 8 § Weka 3.8 for model learning § SentiWordNet for polarity extraction § Stanford POS tagging library for part-of-speech tagging q 2 preprocessed datasets § minVotes = 12: 41,681 reviews § minVotes = 23: 11,083 reviews § In each dataset, reviewers have between 30 to 50 reviews 19 Te Tests Helpfulness model accuracy is similar to the recent proposals of the q state-of-the-art RMSE: the error between the helpfulness model scores and the actual § helpfulness scores 20 10

4/20/18 Te Tests q Using KTs § a-mKRMSE : error between the KT of the actual helpfulness scores and the KT of the helpfulness as computed with the model § a-AggKRMSE : error between the KT of the actual helpfulness scores and the aggregation of the KT scores of each feature taken independently (sub- skill) 21 Le Lesson ons le learn rned & perspect ctiv ives KT is op(mis(c and has an intrinsic smoothing behavior q Finer skills works be9er than coarser ones q Perspec(ves q Short term § Tes+ng with more helpfulness models and skill acquisi+on models q Understanding be;er the rela+onship between the linear coefficient learned for the q helpfulness model and the KT parameters of the corresponding sub-skills Longer term § Applica+on to other datasets, contexts and skills q Eg , how to assess data explora+on, or how to assess deep learning’s produc+ons § 22 11

4/20/18 Re References q [Corbe' & Anderson, UMAI 1995] § Albert T. Corbe,, John R. Anderson: Knowledge Tracing: Modelling the Acquisi>on of Procedural Knowledge. User Model. User-Adapt. Interact. 4(4): 253-278 (1995) [Wang & Heffernan, AIED 2013] q § Yutao Wang, Neil T. Heffernan: Extending Knowledge Tracing to Allow Par>al Credit: Using Con>nuous versus Binary Nodes. AIED 2013: 181-188 [Hawkins & al., ITS 2014] q § William J. Hawkins, Neil T. Heffernan, Ryan Shaun Joazeiro de Baker: Learning Bayesian Knowledge Tracing Parameters with a Knowledge Heuris>c and Empirical Probabili>es. Intelligent Tutoring Systems2014: 150-155 [He & McAuley, WWW 2016] q § Ruining He and Julian McAuley. 2016. Ups and downs: Modeling the visual evolu>on of fashion trends with one-class collabora>ve filtering. In WWW. 507–517. q [KorfiaNs & al., ECRA 2012] Nikolaos Korfia>s, Elena García-Bariocanal, and Salvador Sánchez-Alonso. 2012. Evalua>ng content quality and § helpfulness of online product reviews: The interplay of review helpfulness vs. review content. Electronic Commerce Research and Applica>ons 11, 3 (2012), 205–217. 23 Q&A Q& 24 12