improving automated feedback
play

Improving Automated Feedback Building a Rule Feedback Generator - PowerPoint PPT Presentation

Dec 16, 2022 •320 likes •1.1k views

Improving Automated Feedback Improving Automated Feedback Building a Rule Feedback Generator Eric Bouwers September 27, 2007 Eric Bouwers Improving Automated Feedback (Generic) Outline The problem 1 Our solution 2 Evaluation 3

  1. Improving Automated Feedback Improving Automated Feedback Building a Rule Feedback Generator Eric Bouwers September 27, 2007 Eric Bouwers

  2. Improving Automated Feedback (Generic) Outline The problem 1 Our solution 2 Evaluation 3 Conclusion 4 Eric Bouwers

  3. Improving Automated Feedback > The problem Procedural skills Eric Bouwers

  4. Improving Automated Feedback > The problem Feedback Eric Bouwers

  5. Improving Automated Feedback > The problem Feedback in Education Eric Bouwers

  6. Improving Automated Feedback > The problem Feedback in Education Eric Bouwers

  7. Improving Automated Feedback > The problem Current solutions Eric Bouwers

  8. Improving Automated Feedback > The problem Current solutions Eric Bouwers

  9. Improving Automated Feedback > The problem Current solutions Eric Bouwers

  10. Improving Automated Feedback > The problem Current solutions Eric Bouwers

  11. Improving Automated Feedback > The problem Current solutions Eric Bouwers

  12. Improving Automated Feedback > The problem Current solutions Eric Bouwers

  13. Improving Automated Feedback > The problem Current solutions Eric Bouwers

  14. Improving Automated Feedback > The problem Current solutions Eric Bouwers

  15. Improving Automated Feedback > The problem Synopsis Feedback is important in learning Personal feedback is not feasible without (computerized) help Current solutions are either: Only Correct/Incorrect solutions Result of extensive research Eric Bouwers

  16. Improving Automated Feedback > Our solution Goal Research question How can we make the generation of high-quality, domain-specific feedback easier? Eric Bouwers

  17. Improving Automated Feedback > Our solution Goal Research question How can we make the generation of high-quality, domain-specific feedback easier? Basic idea Create a generic framework which separates the knowledge of rule-feedback generation from knowledge about the domain. Eric Bouwers

  18. Improving Automated Feedback > Our solution Requirements Should always be able to produce basic feedback Eric Bouwers

  19. Improving Automated Feedback > Our solution Requirements Should always be able to produce basic feedback More detailed/complete input leads to better feedback Eric Bouwers

  20. Improving Automated Feedback > Our solution Requirements Should always be able to produce basic feedback More detailed/complete input leads to better feedback Can be instantiated on different domains (with little effort) Eric Bouwers

  21. Improving Automated Feedback > Our solution Requirements Should always be able to produce basic feedback More detailed/complete input leads to better feedback Can be instantiated on different domains (with little effort) Adaptable to a single class-room (or student!) Eric Bouwers

  22. Improving Automated Feedback > Our solution Overall design Eric Bouwers

  23. Improving Automated Feedback > Our solution Overall design Phase 1: Input: CT + PT Output: Correct/Incorrect message Eric Bouwers

  24. Improving Automated Feedback > Our solution Overall design Phase 2: Input: CT + PT + Student rule Output: Correct/Incorrect message-tuple Eric Bouwers

  25. Improving Automated Feedback > Our solution Overall design Phase 3: Input: CT + PT + Allowed rules Output: Configurable message, phase level Eric Bouwers

  26. Improving Automated Feedback > Our solution Overall design Phase 3: Example Message Input: Unfortunately, this step is not correct. CT + PT I think you wanted to apply this rule: [RULE]. + Allowed rules This would result in [PTAPPLIEDRULE]. Can you see the difference with [CT]? Output: Configurable message, phase level Eric Bouwers

  27. Improving Automated Feedback > Our solution Overall design Phase 4: Input: CT + PT + Allowed rules + Buggy rules Output: Configurable message, rule level Eric Bouwers

  28. Improving Automated Feedback > Our solution First phase Input: PT, CT Output: Correct or Incorrect Algorithm: Solve both terms and check whether their results are semantically equal. Eric Bouwers

  29. Improving Automated Feedback > Our solution First phase Input: PT, CT Output: Correct or Incorrect Algorithm: Solve both terms and check whether their results are semantically equal. Implementation firstPhase :: RFG a => RFGSolver a -> RFGEqual a -> a -> a -> String firstPhase solve equal pt ct = let resultCt = solve ct resultPt = solve pt in if equal resultPt resultCt then getConfig Correct else getConfig Incorrect Eric Bouwers

  30. Improving Automated Feedback > Our solution First phase Input: PT, CT Output: Correct or Incorrect Algorithm: Solve both terms and check whether their results are semantically equal. Implementation firstPhase :: RFG a => RFGSolver a -> RFGEqual a -> a -> a -> String firstPhase solve equal pt ct = let resultCt = solve ct resultPt = solve pt in if equal resultPt resultCt then getConfig Correct else getConfig Incorrect Eric Bouwers

  31. Improving Automated Feedback > Our solution First phase Input: PT, CT Output: Correct or Incorrect Algorithm: Solve both terms and check whether their results are semantically equal. Implementation firstPhase :: RFG a => RFGSolver a -> RFGEqual a -> a -> a -> String firstPhase solve equal pt ct = let resultCt = solve ct resultPt = solve pt in if equal resultPt resultCt then getConfig Correct else getConfig Incorrect Eric Bouwers

  32. Improving Automated Feedback > Our solution Third phase Input: PT, CT and allowed rules Output: Configurable message Eric Bouwers

  33. Improving Automated Feedback > Our solution Third phase Input: PT, CT and allowed rules Output: Configurable message Algorithm Given two terms, calculate the rewrite rule δ between these terms. Determine which allowed rule is closest to δ . Apply rules from the set to the PT if possible and recurse. The result is the rule which is a closest match. Eric Bouwers

  34. Improving Automated Feedback > Our solution Third phase Input: PT, CT and allowed rules Output: Configurable message Algorithm Given two terms, calculate the rewrite rule δ between these terms. Determine which allowed rule is closest to δ . Apply rules from the set to the PT if possible and recurse. The result is the rule which is a closest match. Eric Bouwers

  35. Improving Automated Feedback > Our solution Third phase Input: PT, CT and allowed rules Output: Configurable message Algorithm Given two terms, calculate the rewrite rule δ between these terms. Determine which allowed rule is closest to δ . Apply rules from the set to the PT if possible and recurse. The result is the rule which is a closest match. Eric Bouwers

  36. Improving Automated Feedback > Our solution Calculating rewrite rules PT = 1 + 5 + 3 CT = 3 + 6 Eric Bouwers

  37. Improving Automated Feedback > Our solution Calculating rewrite rules PT = 1 + 5 + 3 CT = 3 + 6 Eric Bouwers

  38. Improving Automated Feedback > Our solution Calculating rewrite rules PT = 1 + 5 + 3 CT = 3 + 6 Eric Bouwers

  39. Improving Automated Feedback > Our solution Calculating rewrite rules PT = 1 + 5 + 3 CT = 3 + 6 Eric Bouwers

  40. Improving Automated Feedback > Our solution Calculating rewrite rules PT = 1 + 5 + 3 CT = 3 + 6 Non-Equal ([1,5,3],[3,6, ]) Eric Bouwers

  41. Improving Automated Feedback > Our solution Calculating rewrite rules PT = 1 + 5 + 3 CT = 3 + 6 Equal, Associative ([1,5,3],[3,6, ]) , ([1,5,3],[ ,3,6]) Eric Bouwers

  42. Improving Automated Feedback > Our solution Calculating rewrite rules PT = 1 + 5 + 3 CT = 3 + 6 Equal, Associative and Commutative ([1,5,3],[3,6, ]), ([1,5,3],[ ,3,6]), ([1,5,3],[6,3, ]), ([1,5,3],[ ,6,3]), ([1,3,5],[3,6, ]), ([1,3,5],[ ,3,6]), ([1,3,5],[6,3, ]), ([1,3,5],[ ,6,3]), ([3,1,5],[3,6, ]), ([3,1,5],[ ,3,6]), ([3,1,5],[6,3, ]), ([3,1,5],[ ,6,3]), ([3,5,1],[3,6, ]), ([3,5,1],[ ,3,6]), ([3,5,1],[6,3, ]), ([3,5,1],[ ,6,3]), ([5,3,1],[3,6, ]), ([5,3,1],[ ,3,6]), ([5,3,1],[6,3, ]), ([5,3,1],[ ,6,3]), ([5,1,3],[3,6, ]), ([5,1,3],[ ,3,6]), ([5,1,3],[6,3, ]), ([5,1,3],[ ,6,3]) Eric Bouwers

  43. Improving Automated Feedback > Our solution Calculating rewrite rules PT = 1 + 5 + 3 CT = 3 + 6 Equal, Associative and Commutative ([1,5,3],[3,6, ]), ([1,5,3],[ ,3,6]), ([1,5,3],[6,3, ]), ([1,5,3],[ ,6,3]), ([1,3,5],[3,6, ]), ([1,3,5],[ ,3,6]), ([1,3,5],[6,3, ]), ([1,3,5],[ ,6,3]), ([3,1,5],[3,6, ]), ([3,1,5],[ ,3,6]), ([3,1,5],[6,3, ]), ([3,1,5],[ ,6,3]), ([3,5,1],[3,6, ]), ([3,5,1],[ ,3,6]), ([3,5,1],[6,3, ]), ([3,5,1],[ ,6,3]), ([5,3,1],[3,6, ]), ([5,3,1],[ ,3,6]), ([5,3,1],[6,3, ]), ([5,3,1],[ ,6,3]), ([5,1,3],[3,6, ]), ([5,1,3],[ ,3,6]), ([5,1,3],[6,3, ]), ([5,1,3],[ ,6,3]) Eric Bouwers

  44. Improving Automated Feedback > Our solution Calculating rewrite rules PT = 1 + 5 + 3 CT = 3 + 6 Result: ([1,5],[6]) Eric Bouwers

  45. Improving Automated Feedback > Our solution Calculating rewrite rules PT = 1 + 5 + 3 CT = 3 + 6 Result: δ = 1 + 5 ⇒ +6 Eric Bouwers

  46. Improving Automated Feedback > Our solution Calculating rewrite rules PT = 1 + 5 + 3 CT = 3 + 6 Result: δ = 1 + 5 ⇒ 6 Eric Bouwers

  47. Improving Automated Feedback > Our solution Defining the distance Distance between rules: Eric Bouwers

Recommend

Improving Automated Email Tagging with Implicit Feedback Mohammad S. Sorower Michael Slater

Improving Automated Email Tagging with Implicit Feedback Mohammad S. Sorower Michael Slater

November 9, 2015 Improving Automated Email Tagging with Implicit Feedback Mohammad S. Sorower Michael Slater Thomas G. Dietterich OUTLINE Motivation The Email Predictor EP2 Instrumentation Algorithms Baseline Algorithms

232 views • 20 slides
Exploring the use of computational linguistics for automated formative feedback in the Humanities

Exploring the use of computational linguistics for automated formative feedback in the Humanities

Exploring the use of computational linguistics for automated formative feedback in the Humanities University of Edinburgh Jessie Paterson, Christian Lange, Iqbal Akhtar, Francisco Iacobelli, Paul Anderson, Annette Leonhard Contact:

313 views • 17 slides
Measuring and improving quality of automated program repair

Measuring and improving quality of automated program repair

Measuring and improving quality of automated program repair Yuriy Brun, UMass Amherst Ted Smith

651 views • 22 slides
Exploring the Integration of User Feedback in Automated Testing of Android Applications G.

Exploring the Integration of User Feedback in Automated Testing of Android Applications G.

Exploring the Integration of User Feedback in Automated Testing of Android Applications G. Grano, A. Ciurumelea, S. Panichella, F. Palomba, H. Gall SANER 2018, 20-23 March, Campobasso (Italy) grano@ifi.uzh.ch giograno90 149 billions of

593 views • 27 slides
Improving the Last Stages of Life Preliminary Feedback from Law Reform Consultations in Ontario

Improving the Last Stages of Life Preliminary Feedback from Law Reform Consultations in Ontario

Improving the Last Stages of Life Preliminary Feedback from Law Reform Consultations in Ontario Ryan Fritsch, Project Lead ICEL2 Conference Halifax September 2017 Law Commission of Ontario LCOs Improving Last Stages of Life Project

258 views • 25 slides
Improving Test Case Generation for Web Applications Using Automated Interface Discovery William

Improving Test Case Generation for Web Applications Using Automated Interface Discovery William

Improving Test Case Generation for Web Applications Using Automated Interface Discovery William G.J. Halfond and Alessandro Orso Georgia Institute of Technology Web Application Overview Request Web http://host?login=alice&pin=1234

892 views • 40 slides
Improving the Efficiency of Manual Ground Truth Labeling Using Automated Anatomy Segmentation

Improving the Efficiency of Manual Ground Truth Labeling Using Automated Anatomy Segmentation

Improving the Efficiency of Manual Ground Truth Labeling Using Automated Anatomy Segmentation Hongzhi Wang PhD, Prasanth Prasanna MD, Jose Morey MD, Tanveer F. Syeda-Mahmood PhD Medical Sieve Group, IBM Almaden Research Center Anatomy

469 views • 18 slides
Ask-Elle An Adaptable Programming Tutor for Haskell Giving Automated Feedback Bastiaan Heeren

Ask-Elle An Adaptable Programming Tutor for Haskell Giving Automated Feedback Bastiaan Heeren

Ask-Elle An Adaptable Programming Tutor for Haskell Giving Automated Feedback Bastiaan Heeren April 26, 2016 OU Research Seminar 2. exercise description 4. high-level hint 5. bottom-out hint 3. student program 1. list of exercises Why use

727 views • 33 slides
Lowering the Operational Costs through Improving Crew Readiness and Automated Real-time Decision

Lowering the Operational Costs through Improving Crew Readiness and Automated Real-time Decision

Lowering the Operational Costs through Improving Crew Readiness and Automated Real-time Decision support using Digital Twin eDrilling a world leading supplier of AI, machine learning, and predictive analytics solutions to the oil and gas

754 views • 14 slides
Teaching Software Testing with Automated Feedback James Perretta and Andrew DeOrio, University of

Teaching Software Testing with Automated Feedback James Perretta and Andrew DeOrio, University of

Teaching Software Testing with Automated Feedback James Perretta and Andrew DeOrio, University of Michigan ASEE Annual Conference and Exposition, June 2018 1 How important is it for your students to learn software testing? 2 How do your

542 views • 25 slides
Improving eLearning through Statistical Feedback Bachelors Thesis Final Talk Student:

Improving eLearning through Statistical Feedback Bachelors Thesis Final Talk Student:

Improving eLearning through Statistical Feedback Bachelors Thesis Final Talk Student: Beatris Burdeva Supervisor: Prof. Dr.-Ing. Georg Carle Advisers: Marc-Oliver Pahl, Stefan Liebald Garching, 10. July 2017 Overview I. Motivation II.

783 views • 26 slides
Improving a Neural Semantic Parser by Counterfactual Learning from Human Bandit Feedback Carolin

Improving a Neural Semantic Parser by Counterfactual Learning from Human Bandit Feedback Carolin

Introduction Task Objectives Experiments Conclusion Improving a Neural Semantic Parser by Counterfactual Learning from Human Bandit Feedback Carolin Lawrence , Stefan Riezler Heidelberg University Institute for Computational Linguistics

430 views • 18 slides
Presentation and Feedback Policy Introduction We are committed to a positive approach to providing

Presentation and Feedback Policy Introduction We are committed to a positive approach to providing

Presentation and Feedback Policy Introduction We are committed to a positive approach to providing feedback and improving childrens work at Alexandra Junior School , where childrens achieveme nts are praised and valued. Feedback offers the

669 views • 7 slides
Dirty Tests Practise improving brittle, complicated, incomprehensible automated tests The case

Dirty Tests Practise improving brittle, complicated, incomprehensible automated tests The case

Dirty Tests Practise improving brittle, complicated, incomprehensible automated tests The case of the Time Dependent Test The case of the Obscure Test Background Transport Information System Assign Carrier to Transfer Shipment of Goods

740 views • 39 slides
Evaluation or check ? Evaluation purpose? Feedback from exercise-lessons or best practise

Evaluation or check ? Evaluation purpose? Feedback from exercise-lessons or best practise

Evaluation or check ? Evaluation purpose? Feedback from exercise-lessons or best practise Test the performance/capacity. Improving manuals-implementing result Improving the evaluation itself Goals from the old HELCOM

189 views • 16 slides
VetTag: improving automated veterinary diagnosis coding via large-scale language modeling

VetTag: improving automated veterinary diagnosis coding via large-scale language modeling

VetTag: improving automated veterinary diagnosis coding via large-scale language modeling Published in npj (Nature) Digital Medicine Yuhui Zhang, Allen Nie, James Zou Introduction Large-scale veterinary clinical records can become a powerful

529 views • 14 slides
Automated Reasoning: Some Successes and New Challenges Predrag Jani ci c

Automated Reasoning: Some Successes and New Challenges Predrag Jani ci c

What is this talk about? What is automated reasoning? Automated reasoning in propositional logic Automated reasoning in first-order logic Automated reasoning in higher-order logic Automated reasoning in geometry Conclusions Automated

806 views • 52 slides
Towards an Automated Fault Localizer while Designing Meta-models Adel Ferdjoukh and Jean-Marie

Towards an Automated Fault Localizer while Designing Meta-models Adel Ferdjoukh and Jean-Marie

Towards an Automated Fault Localizer while Designing Meta-models Adel Ferdjoukh and Jean-Marie Mottu MDEbug@MODELS 2018, Copenhagen (Kbenhavn) null Motivation Automated fault localization Tooling Ideas for improving Conclusion Synopsis 1

586 views • 25 slides
4303 Ed/Serious Feedback and Reinforcement Feedback and Reinforcement Both of them are

4303 Ed/Serious Feedback and Reinforcement Feedback and Reinforcement Both of them are

4303 Ed/Serious Feedback and Reinforcement Feedback and Reinforcement Both of them are the most important concepts in learning. Any differences? Feedback Feedback involves providing learners with information about their

710 views • 24 slides
Feedback EEG Brain Feedback EEG Brain Feedback Tends to be Supposed

Feedback EEG Brain Feedback EEG Brain Feedback Tends to be Supposed

Feedback EEG Brain Feedback EEG Brain Feedback Tends to be Supposed over-simulated to be calm Supposed to Tends to be be alert under- stimulated Memory Stroke problems Seizure Head Sleep disorders injury disorders

465 views • 33 slides
Improving code reuse in clang tools with clangmetatool Daniel Ruoso druoso@bloomberg.net

Improving code reuse in clang tools with clangmetatool Daniel Ruoso druoso@bloomberg.net

Improving code reuse in clang tools with clangmetatool Daniel Ruoso druoso@bloomberg.net Bloomberg October 17, 2018 Static Analysis and Automated Refactoring at Bloomberg Static Analysis and Automated Refactoring at Bloomberg

1.13k views • 44 slides
Automated Connected - Mobile Strategies & Actions towards Automated & Connected

Automated Connected - Mobile Strategies & Actions towards Automated & Connected

Automated Connected - Mobile Strategies & Actions towards Automated & Connected Driving in Austria Action Plan Automated Connected Mobile Framework, Visibility & Evidence Competences Use Cases & Value Creation &

655 views • 17 slides
AUTOMATED TRAFFIC ENFORCEMENT March 25, 2019 Overview Request for Decision March 18,

AUTOMATED TRAFFIC ENFORCEMENT March 25, 2019 Overview Request for Decision March 18,

AUTOMATED TRAFFIC ENFORCEMENT March 25, 2019 Overview Request for Decision March 18, 2019 COW report and subsequent feedback from Council Strong desire to provide clear direction to administration and community Motion as

343 views • 13 slides
Announcements: - Thank you for your course feedback! - Watch out for homework 2 feedback poll

Announcements: - Thank you for your course feedback! - Watch out for homework 2 feedback poll

Announcements: - Thank you for your course feedback! - Watch out for homework 2 feedback poll - Course project TAs will reach out with feedback - Regrade requests for HW1 Deadline Thu next week at 23:59pm - Today: HW2 due / HW3

753 views • 61 slides

More recommend