Diagrams, Interfaces, and Klingons Based on Kieras & Bovair's - PDF document

Slides presented at the ONR, NSF, and DARPA Symposium on Reasoning and Learning in Cognitive Systems, CSLI, Stanford, March 20-21, 2004. Modelling How and When Learning Happens in a Diagrammatic Reasoning Task Frank E. Ritter School of Information Sciences and Technology, Psychology, and Computer Science Penn State Peter A. Bibby School of Psychology U. of Nottingh a m 50 120 Subject Problem Solving Time Soar Model 100 Decision Cycles 40 80 30 60 20 40 10 20 0 0 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 Supported by Joint Council Initiative in HCI and Cognitive Science 1 21/3/04 Diagrams, Interfaces, and Klingons ■ Based on Kieras & Bovair's Starship+ (='Klingon’) interface MAIN ACCUMULATOR ma ENERGY ENERGY LASER BOOSTER BOOSTER BANK ONE TWO sa SECONDARY ACCUMULATOR eb1 eb2 ONE sa1 off on sa2 SECONDARY ACCUMULATOR POWER TWO SOURCE Device Schematic 2 21/3/04

Task Details ■ Subjects are given: (i) a general intro. to the problem (ii) basic information on the interface (iii) a diagram of the underlying circuit ■ Subjects are told that ONE component in the circuit is faulty, and are asked to indicate which o n e 3 21/3/04 Some Example Faults PS EB1 EB2 MA SA1 SA2 LB PS EB1 EB2 MA SA1 SA2 LB off off off off off off off on eb1 eb2 sa1 sa2 sa ma off on eb1 eb2 sa1 sa2 sa ma MAIN ACCUMULATOR PS EB1 EB2 MA SA1 SA2 LB ma ENERGY ENERGY LASER BOOSTER BOOSTER BANK ONE TWO sa SECONDARY ACCUMULATOR eb1 eb2 ONE off off off sa1 off on eb1 eb2 sa1 sa2 sa ma off on sa2 SECONDARY ACCUMULATOR POWER TWO SOURCE Device Schematic 4 21/3/04

Why pay attention? ■ How to learn with interaction ■ Illustrates human-level learning ➤ Forbus gold standard, ➤ Langley notes “just on this task” ■ Shows that Soar chunker models learning and transfer of learning ■ Shows learning and/with reasoning ■ There is no wireless network 5 21/3/04 Diag-Soar (v16 ) � (R � � � itter & Bibby, 2 0 01 ) ■ 173 rules + 220 chunks (new rules) ■ Schematic knowledge represented a s linked lists, organized as 'routes' through the circuit ■ Visual interface information represented as declarative structures for lights & switches ■ Status of interface diagram represented in top goal, accessed by attend 6 21/3/04

Diag-Soar’s Problem Spaces OLVE - PROBLEM � Rit ter, 2 0 01. Soar � � � S ATTEND diagnose ICCM tutorials COMPREHEND R EPORT ( SOLVE - PROBLEM ) ( COMPR EHEND ) CH OOSE - COMPONENT find- TE ST - COMPONENT fault R ES ET ( TE ST - COMPONENT ) ( CH OOSE - COMPONENT ) CHECK - LIT CHECK - SWITCH - DIAG test- DIAG - CHECK - SWITCH - DECIDE comp S U GGE STION CHECK - PRE VIOUS - DIAG select INTERFACE - comp CHECK - PRE VIOUS - DE CIDE S U GE STION DECIDE - STATUS ( DECIDE - ( INTER FACE STATUS ) ( CHECK - LIT ) ( DIAG - S U GG .) - SU GG .) ( CHECK - SWITCH ) ( CHECK - PRE VIOUS ) interf.- ALITY - CHECK RE DIAG - CH OICE select INTERF . check-world diag.-select E - SWITCH CH OOS CH OICE E - PRE CH OOS VIOUS Diag-Soar (cont.) ■ Organization of components on the interface diagram 'sequences' the checking of each component ■ If a subgoal requires perceptual information, goal stack must be re-built ■ attend and comprehend operators used to represent the perceptual components of SO LVE - PROBLEM diagnose ATTEND the task CO MPRE HEND REP ORT ( SOLVE - P OBLEM ) R ( CO MPRE HEND ) CHO OSE - COMPO NENT find- TEST - COM P ONENT fault RESET ( CHO OSE - ( TEST - COM P ONENT ) NENT ) COMPO CHECK - LIT CHECK - SWITCH - DIAG DIAG - test- CHECK - SWITCH - DECIDE SUG G ESTION comp CHECK - PRE VIOUS - DIAG select comp INTERFACE - CHECK - PRE VIOUS - DECIDE SUG ESTION DECIDE - STATUS ( DECIDE - ( INT ERFACE STATUS ) ( CHECK - LIT ) - SU GG .) ( DIAG - SU GG .) ( CHECK - SWITCH ) ( CHECK - PR EVIOUS ) interf.- REALITY - CHECK 8 DIAG - CH OIC E 21/3/04 select INTERF . check-world diag.-select CHO OSE - SWITCH CHO ICE

Representation? ■ Experience and knowledge stored in Soar production rules ■ Rules organized as implementing problem spaces, operators, and choices between them ■ All learned knowledge is also SOLVE - P R OBLEM represented in rules with ATT END diagnose COM PRE HEND REP ORT ( SOLVE - PRO BLE M ) ( COM PRE HEND ) CH OO SE - CO MPONENT find- T E S T - COMPO NENT fault these structures RESET ( T E T - COMPO NENT ) ( CH OO SE - S COMPO NENT ) CHECK - LIT CHECK - SWITCH - DIAG DIAG - test- CHECK - SWITCH - DECIDE SU GGE S T IO N comp select CHECK - P REVIOUS - DIAG ACE - comp INTERF CHECK - P REVIOUS - DECIDE SU GESTION DECIDE - STATUS ( DECIDE - ( INT ERFACE STATUS ) ( CHECK - LIT ) ( DIA G - SUG G .) - SU GG .) ( CHECK - SWITCH ) ( CHECK - PRE VIO US ) interf.- REALITY - C select DIA G - CH OIC E INTERF . check-world diag.-select CH OOSE - SWITCH 9 CHO ICE 21/3/04 CH OOSE - PRE VIOUS Soar, Learning in Action Add 1+2 New rule: Result=3 If op is add 1+2 then result = 3 Op No Change Count problem space 10 21/3/04

� Tor et al. 2004 CaDaDis, BRIMS What is Learned? ■ (All learning implemented as new rules) ■ Where to look - implementation of Choose-component; and creation of Attend and additions to Attend ■ What stimuli mean - implementation of Comprehend (and lower operations) ■ Augmentations to the state from previous problem solving ■ Huge amounts of transfer 12 21/3/04

General Results ■ Does the task ■ General Strategy — sequential components checking — emerges from the interface representation , consistent with Ss protocols ■ Final ('fully chunked') behaviour reflects this strategy, behaving as if it were simply a recognition (immediate response) task 13 21/3/04 Average RTs by Subjects 16 80 14 70 Problem Solving Time (s) 12 60 Model Cycles 10 50 8 40 6 30 4 20 Problem Solving Time (s) 2 10 Model Cycles 0 0 1 2 3 4 5 6 7 8 9 10 Subject 14 21/3/04

The data ■ 10 subjects solved ■ 20 problems ■ Each subject saw a different series of problems ■ Problems sampled with replacement (nominally) ■ RTs and answers recorded ■ Incorrect trials discarded ■ Some comments taken at end of trials 15 21/3/04 RTs by Fault Type ■ Accounts for 90% of Ss' RT variability 22 140 Problem Solving 20 Time (s) 120 18 Problem Solving Time (s) Model 16 Cycles 100 Decision Cycles 14 80 12 10 60 8 40 6 4 20 2 0 0 PS EB1 EB2 MA SA1 SA2 LB Fault 16 21/3/04

RTs by Trial Number 240 34 Problem Solving Time (s) 220 30 Problem solving time (s) Model Cycles 200 26 180 Model cycles 160 22 140 18 120 100 14 80 10 60 6 40 20 2 0 2 4 6 8 10 12 14 16 18 20 Trial 17 21/3/04 Matches RT by Individual’s Trials (s9) 30 200 28 Problem Solving Time (s) 180 26 Problem Solving Time (s) 24 160 Model Cycles 22 Model Cycles 140 20 18 120 16 100 14 12 80 10 60 8 6 40 4 20 2 0 0 PS EB2 SA2 LB LB MA SA2 SA1 LB PS MA SA1 SA2 EB1 MA EB2 SA1 EB1 EB1 Fault 18 21/3/04

Learns like No Other Model Tested with Data (!?) ∴ No model 1 with automatic learning, tested in detail across tasks while learning ■ Able does not do transition on the fly ■ Anderson's tutors just add rules ■ VanLehn looked at new rule acquisition but added rules by hand 1 Altmann (‘99) programming model does this with recognition of objects 19 21/3/04 Data and Regularities Left Behind ■ Modelling the perceptual improve- ments in motor skills omitte d ■ No account of initial learning of the task ■ Model accounts for < 10% of the variability for some subjects (2 out of 10) ■ 2nd trials on a problem are too fast in the model; Verbal protocols not reported her e 20 21/3/04

Evidence for Reflection in Participants Trial 3 Trial 5 21 21/3/04 What Can Diag-Soar Tell Us About Reasoning with Diagrams ? ■ Our subjects learn and only learn the diagram information that is relevant to the context of each stage of the problem-solving ■ The model supports that subjects still use the diagram information in a cyclical, iterative fashion as an external resource to support the problem-solving sequence ■ Result is recognition-driven problem solving, rather than model-driven behaviour arising out of problem solving ■ Soar’s chunking models transfer in this task 22 21/3/04