the robotics design lab Peer- Review Easily with Confidence : A Look - PDF document

1/09/2017 the robotics design lab Peer- Review Easily with “Confidence” : A Look at Replication and Randomization as tools for Statistically-Significant Assessment via Platypus UQ Robotics 2 1

1/09/2017 UQ Robotics 3 What is Platypus? Many Things Put Together! • (Yet another) Electronic Submission System • A Question-based Online Peer Assessment Tool • An “Anonymous” Review + Student Progress Tool • A Tool With a Nuanced (Data-centric) View of Assessment UQ Robotics 4 2

1/09/2017 Question-Based Peer Review: Workflow Model I. Collect Assignments (one per student) A(1) - Q(1) A(2) - Q(1) A(M) - Q(1) A(1) - Q(2) A(2) - Q(2) A(M) - Q(2) . . . . . . . . . . . . A(1) – Q(N) A(2) – Q(N) A(M) – Q(N) Assignment 1 Assignment 2 Assignment M II. Randomly Shuffle Questions between assignments to create kM “Papers” (Where k is the peer review factor, or the number of papers a student needs to review, eg 3) A(1) - Q(1) A(M) - Q(1) A(2) - Q(2) A(1) - Q(2) . . . A(M) – Q(N) A(2) – Q(N) Paper 1 Paper kM III. Each student grades k papers (may be assigned randomly or via an alternative metric) UQ Robotics 5 Why a Question-Based Shuffle? • Replication & Randomization A(1) - Q(1) • Assessment is an Observer A(2) - Q(2) ∴ We would like N (0, σ) A(M) – Q(N) • Less Numbers Needed for Analysis Paper 1 – A Project Class is not Super Large A(M) - Q(1) • Summative Assessment by parts A(1) - Q(2) • Faster feedback A(2) – Q( N) Paper kM UQ Robotics 6 3

1/09/2017 Platypus UQ Robotics 7 Platypus UQ Robotics 8 4

1/09/2017 It’s Open… UQ Robotics 9 SQL + Laravel UQ Robotics 10 10 5

1/09/2017 SQL: Sometimes it’s handy SELECT users.`first_name`, users.`last_name`, users.`student_id`, users.`id` FROM ((users INNER JOIN answers ON users.`id` = answers.`user_id`) INNER JOIN text_blocks ON text_blocks.`id` = answers.`answer_text_id`) WHERE text_blocks.text LIKE '%fbcdn.net%' UQ Robotics 11 11 Benefit: Focus on Learning Outcomes UQ Robotics 12 12 6

1/09/2017 Peer Feedback Research UQ Robotics 13 13 Question-Based Peer Review: Where it Works & Doesn’t UQ Robotics 14 14 7

1/09/2017 “Statistics”: Tools to “Assess” Learning? UQ Robotics 15 15 Basic Counts… • Scores Histograms… • Grade Correlations… UQ Robotics 16 16 8

1/09/2017 Interesting Finds: Dunning-Kruger Effect? • Correlation Between Students' Actual & Estimated & Final Grades UQ Robotics 17 17 Trustworthiness of Peer & Tutor Marking UQ Robotics 18 18 9

1/09/2017 Longer answers better? • Total Student Feedback Output Characters vs. Total Student Marks Received UQ Robotics 19 19 Longer feedback better? • Tutor Feedback Length and Final Course Grade UQ Robotics 20 20 10

1/09/2017 Positive Δ : Positive Outcome? UQ Robotics 21 21 Worldwide Access UQ Robotics 22 22 11

1/09/2017 Future Work • Richer Inputs – [Matlab] Figures – Simple Diagrams/Sketches  Rich media in general • Alternate Question Types – Lab Reports, etc.  User-Defined? • Team Submissions/Review F( x ) • Visualization Tools UQ Robotics 23 23 Will it Replace the Teacher? Today we piloted Gradescope 's AI-assisted grading. So here we are, CS 188 all done with 650 18-page exams after just three hours of grading! AI graded AI  UQ Robotics 24 24 12

1/09/2017 Platypus: Available For Your Class Too 🌐 http://openplatypus.org/ UQ Robotics 25 25 Students will be students UQ Robotics 26 26 13

1/09/2017 Collaboration: More than Just for Robots UQ Robotics 27 27 The Goal: Create Change  UQ Robotics 28 28 14

1/09/2017 Does Your Job Make the World a Better Place? Source: A. McCann, BusinessWeek, 8/8/13 UQ Robotics 29 29 For More Information: 🌐 🌐 http://openplatypus.org/ UQ Robotics 30 30 15

1/09/2017 UQ Robotics: Dynamic Systems in Motion Diverse international Aerial Robotics research group Hanna Kurniwati (NUS/MIT) Paul Pounds (ANU/Yale) Surya Singh (Stanford/Syd) Systems Planning Under Uncertainty & Decision-Making Mechanics of Motion UQ Robotics 31 31 UQ Robotics 32 32 16