Using Inverse Planning for Personalized Feedback Anna N. Rafferty - PowerPoint PPT Presentation

Using Inverse Planning for Personalized Feedback Anna N. Rafferty Computer Science Department, Carleton College Rachel A. Jansen Thomas L. Griffiths Department of Psychology, University of California, Berkeley

Using Data for Personalization Provide experience X ? Algorithm

Outline • Inverse planning: Diagnosing misunderstandings about equation solving • Developing personalized feedback based on diagnosis • Testing effectiveness of personalized feedback • Future directions

Interpreting Equation Solving: Bayesian Inverse Planning Algebra Algebra skills ( 𝜄 1 ) skills ( 𝜄 2 ) Θ = space of possible understandings p ( θ | equations)

Representing Understanding: Θ θ ∈ Θ : 6-dimensional vector of parameters related to skill 1+3x => 4x Conceptual Mal-rules 3(2+5x) => 6+5x 1+5.9x+3.2x => 1+8.1x Arithmetic -3+5+x => -2+x Planning 3x+5x+4 = 2 => 3x+4 = -5x+2 e.g., Sleeman, 1984; Payne & Squibb, 1990; Koedinger & MacLaren,1997

Bayesian Inverse Planning Algebra Algebra skills ( 𝜄 1 ) skills ( 𝜄 2 ) p ( θ | equations)

Bayesian Inverse Planning Algebra Algebra skills ( 𝜄 1 ) skills ( 𝜄 2 ) p ( θ | equations) ∝ p ( θ ) p (equations | θ ) { { Prior Likelihood Prior : Encode information about what misunderstandings are common

Bayesian Inverse Planning Algebra Algebra skills ( 𝜄 1 ) skills ( 𝜄 2 ) p ( θ | equations) ∝ p ( θ ) p (equations | θ ) { { Prior Likelihood Likelihood : What is the probability of the observed data if the learner has a particular understanding?

Generative Model of Equation Solving: Markov Decision Processes Move 2 to Combine 6 Divide both right side and 2 sides by 3 2 + 3x = 6 3x = 6 + 2 3x = 8 ... 𝜄 affects what actions are considered and transition probabilities for actions.

How are Actions Chosen? Move 2 to Combine 6 Divide both right side and 2 sides by 3 2 + 3x = 6 3x = 6 + 2 3x = 8 ... Assume a noisily optimal policy: p ( a | s ) ∝ exp( θ β · Q ( s, a )) Long term expected value: ! X X p ( s 0 | s, a ) p ( a 0 | s 0 ) Q ( s 0 , a 0 ) Q ( s, a ) = R ( s, a ) + γ s 0 2 S a 0 2 A

Inverse Planning Overview Arithmetic Error Arithmetic Error Parameter Parameter Arithmetic error parameter 5 + 9 = 6 . 0 x + 2 . 0 x + 10 . 0[1 + 1 + 7 . 0 x ] 5 + 9 = 6 . 0 x + 2 . 0 x + 10 + 10 + 70 . 0 x 1 5 + 9 = 6 . 0 x + 2 . 0 x + 20 + 70 . 0 x Distributive property 14 = 6 . 0 x + 2 . 0 x + 20 + 70 . 0 x 5 + 9 = 6 . 0 x + 2 . 0 x + 10 . 0[1 + 1 + 7 . 0 x ] error parameter Probability Probability 5 + 9 = 6 . 0 x + 2 . 0 x + 10 + 10 + 70 . 0 x 5 + 9 = 6 . 0 x + 2 . 0 x + 20 + 70 . 0 x Move term error 14 = 6 . 0 x + 2 . 0 x + 20 + 70 . 0 x 0.5 14 = 76 . 0 x + 2 . 0 x + 20 . 0 parameter 14 = 78 . 0 x + 20 . 0 14 + − 20 = 78 . 0 x − 7 = 78 . 0 x Action planning parameter − 7 78 = 1 x 0 0 0.5 1 . Value Value . . Representation of Model of equation Infer posterior understanding solving as a probability over Θ ( Θ ) (parameterized) (MCMC) MDP

Output for One Learner Move Combine Divide Distributive Planning Arithmetic Probability 0.5 Probability 0.5 Probability 0.5 Probability 0.5 Probability 0.5 Probability 0.5 0 0 0 0 0 0 0 0.5 1 0 1 0 0.5 1 0 0.5 1 0 2 4 0 0.5 1 Value Value Value Value Value Value How do we turn this into a feedback activity?

Feedback Activities Overview of skills and assessment Text explanation and video from Khan Academy Targeted practice with fading scaffolding

Testing Personalized Feedback Session 2: Session 1: Session 3: Feedback Activity Website Problem Website Problem Solving and Multiple Solving and Multiple Choice Test Choice Test

Results: Changes in Performance Across Sessions Accuracy Improvements by Time and Condition Accuracy Improvements by Time and Condition Accuracy Improvements by Time and Condition Accuracy Improvements by Time and Condition 24 Before Feedback Before Feedback Before Feedback After Feedback After Feedback After Feedback 18 Score 12 6 0 Targeted Feedback Random Feedback Targeted Feedback Random Feedback Reliable improvement, but no difference in amount of improvement across conditions.

Performance Based on Proficiency Level of Feedback Skill Accuracy Improvements by Time and Level of Skill 24 Before Feedback After Feedback 18 Score 12 6 0 Skill level > 0.85 Skill level < 0.85

Performance Change for Participants with Varying Skill Levels Accuracy Improvements by Time and Condition for Participants with Some Mastered and Some Unmastered Skills 24 Before Feedback After Feedback 18 Score 12 6 0 Random Feedback Targeted Feedback Reliable difference in amount of improvement by condition.

Contributions and Next Steps • Personalization using inverse planning is helpful for learners who struggle with only some skills • Provides an applied metric assessing the algorithm • Next steps: • Greater specificity and more interactivity in feedback • Longer term interventions

Thank you! Contact: Anna Rafferty, arafferty@carleton.edu Acknowledgements: Thank you to students Jonathan Brodie and Sam Vinitsky for programming contributions. Funding : This work is supported by NSF grant number DRL-1420732.

Skill Proficiencies by Participant 1 Proportion of participants 0.8 0.6 0.4 0.2 0 0 1 2 3 4 5 Number of skills with proficiency < 0.85

Markov Decision Processes a 1 a 2 a 3 s 1 s 2 s 3 ... Actions: - move a term - multiply or divide by a constant - combine two terms - distribute a coefficient - stop solving