OVERVIEW: RULE BASED DEFINITIONS BEHAVIOUR RECOGNITION 1. TEMPORAL - PowerPoint PPT Presentation

School of Informatics, University of Edinburgh School of Informatics, University of Edinburgh OVERVIEW: RULE BASED DEFINITIONS BEHAVIOUR RECOGNITION 1. TEMPORAL GROUP : A TRACKED 1. CROWLEY’S REPRESENTATION INDIVIDUAL OR GROUP THROUGH A VIDEO SEQUENCE 2. ROLE RECOGNITION 2. ROLE : HOW A TEMPORAL GROUP 3. SITUATION RECOGNITION PARTICIPATES IN A SITUATION 4. PREPROCESSING FOR SEQUENCE 3. SITUATION : A PARTICULAR RECOGNITION ASSIGNMENT OF ENTITIES TO ROLES 5. SEQUENCE RECOGNITION + RELATIONS BETWEEN THE 6. WHAT’S MISSING ENTITIES ECVision Summer School: 5 - Behaviour Understanding Fisher slide 1 ECVision Summer School: 5 - Behaviour Understanding Fisher slide 2 School of Informatics, University of Edinburgh School of Informatics, University of Edinburgh BROWSE CONTEXT 4. SCENARIO : A PARTICULAR MOVE BROWSE SEQUENCE OF SITUATIONS MOVE 5. CONTEXT : A GRAPH/NETWORK SITUATION ROLES DESCRIBING ALL POSSIBLE SEQUENCES OF SITUATIONS FOR A MOVE WALKER GIVEN SCENARIO BROWSING BROWSER TYPICAL SCENARIOS: MB, MBM, MBMB, BM, . . . ECVision Summer School: 5 - Behaviour Understanding Fisher slide 3 ECVision Summer School: 5 - Behaviour Understanding Fisher slide 4

School of Informatics, University of Edinburgh School of Informatics, University of Edinburgh RULE BASED BEHAVIOUR RECOGNITION USE CROWLEY’S REPRESENTATION SITUATION ROLE ACTIVITY LEVEL FOCUS ON BROWSE CONTEXT MODEL MOVE WALKER MOVEMENT AS EXAMPLE BROWSING BROWSER STATIONARY MOVE BROWSE MOVE ECVision Summer School: 5 - Behaviour Understanding Fisher slide 5 ECVision Summer School: 5 - Behaviour Understanding Fisher slide 6 School of Informatics, University of Edinburgh School of Informatics, University of Edinburgh ROLE RECOGNITION RULES ROLE RECOGNITION RESULTS ROLE RULE BROWSER IN BROWSABLE LOCATION ≥ 20 FRAMES ROLE GROUND CORRECTLY PERCENT WALKER DEFAULT IF NOT IN OTHER ROLES TRUTH LABELED APPLIED TO TRACKED PERSON, MAPS COUNT COUNT ( ROW T , COL T , T ) → ROLE T BROWSER 3584 2481 69.2% WALKER 33502 31783 94.9% GIVES A SEQUENCE OF ROLE CLASSIFICATIONS: ROLE 1 , ROLE 2 , ROLE 3 , . . . ROLE N SEE LABELED VIDEO ALTERNATIVE ROLE LABELS POSSIBLE: BROWSER COULD ALSO BE AN IDLE PERSON, INSPECTOR ECVision Summer School: 5 - Behaviour Understanding Fisher slide 7 ECVision Summer School: 5 - Behaviour Understanding Fisher slide 8

School of Informatics, University of Edinburgh School of Informatics, University of Edinburgh SCENARIO RECOGNITION PREPROCESSING I: ROLE NOISE CAN THE SEQUENCE OF ROLE CLASSIFICATIONS: FILTERING ROLE 1 , ROLE 2 , ROLE 3 , . . . ROLE N BE ASSUME ISOLATED ROLE LABEL IS RECOGNIZED AS A SCENARIO NOISE INSTANCE? SEQUENCE OF SITUATIONS FIT THE EG: ...WWWWWWBWWWWWW... CONTEXT MODEL? W: WALKER, B: BROWSER BROWSE MOVE FILTER TO RELABEL: ROLE T = MOVE MOST COMMON MEMBER ( { ROLE I : VALID SEQUENCE OF SITUATIONS: T − N ≤ I ≤ T + N } ) ( N = 63) M, MB, MBM, MBMB, ... B, BM, BMB, BMBM, ... ECVision Summer School: 5 - Behaviour Understanding Fisher slide 9 ECVision Summer School: 5 - Behaviour Understanding Fisher slide 10 School of Informatics, University of Edinburgh School of Informatics, University of Edinburgh PREPROCESSING II: TEMPORAL GROUPING PREPROCESSING III: GROUP FILTERING OBSERVATION: ROLES DON’T CHANGE QUICKLY, EG. PERSIST FOR AT LEAST 25 OBSERVATION: CAN’T HAVE GROUPS FRAMES (1 SECOND) SMALLER THAN A GIVEN LENGTH (FUNCTION OF ROLE) COMPRESS ROLE LIST TO GROUPS OF CONSECUTIVE IDENTICAL ROLES ACTION: MERGE SHORT GROUPS WITH LONGEST NEIGHBOUR (OR MOST WALK B A = SIMILAR, OR ...) { WALK A , WALK A +1 , . . . WALK B } W A 0 , B B A , T C B , W D C , B ∞ D → W A 0 , B C A , W D C , B ∞ SEQUENCE OF ROLE GROUPS: D C , B ∞ W A 0 , B B A , T C B , W D D ECVision Summer School: 5 - Behaviour Understanding Fisher slide 11 ECVision Summer School: 5 - Behaviour Understanding Fisher slide 12

School of Informatics, University of Edinburgh School of Informatics, University of Edinburgh RECOGNITION INPUTS RECOGNITION ISSUES • SEQUENCE OF ROLE GROUPS: • ERRONEOUS ROLE GROUPS C , B ∞ W A 0 , B C A , W D D • LOOPS IN CONTEXT TRANSITION • TABLE OF ALLOWABLE ROLES IN A DIAGRAM GIVEN SITUATION • MULTIPLE CONTEXTS TO CONSIDER • TRANSITION DIAGRAM FOR EACH • AMBIGUOUS ROLE GROUPS CONTEXT ECVision Summer School: 5 - Behaviour Understanding Fisher slide 13 ECVision Summer School: 5 - Behaviour Understanding Fisher slide 14 School of Informatics, University of Edinburgh School of Informatics, University of Edinburgh RECOGNITION ALGORITHM SUCH THAT: MODIFIED INTERPRETATION TREE ALGORITHM PRUNED COMBINATORIAL SEARCH • ROLE R I OCCURS IN SITUATION S I TREE • ROLE R I FOLLOWS ROLE R I − 1 IN THE ROLE GROUP LIST EACH NODE IN THE TREE AT LEVEL T IS • SITUATION S I IS A VALID SUCCESSOR A PAIR: (SEQUENCE OF ROLES: SITUATION TO S I − 1 IN THE CONTEXT R 1 , R 2 , . . . R T , SEQUENCE OF SITUATIONS: TRANSITION DIAGRAM S 1 , S 2 , . . . S T ) ECVision Summer School: 5 - Behaviour Understanding Fisher slide 15 ECVision Summer School: 5 - Behaviour Understanding Fisher slide 16

School of Informatics, University of Edinburgh School of Informatics, University of Edinburgh WILDCARDS - * SOME ROLE GROUPS MAY BE INCORRECTLY LABELED GIVES AUGMENTED TRANSITION DON’T WANT TO FAIL TO MATCH IF DIAGRAM: JUST A SMALL PROBLEM * * WILDCARD SITUATION MATCHES ANY MOVE BROWSE ROLE GROUP, SATISFYING ALL PREVIOUS CONSTRAINTS * MOVE ECVision Summer School: 5 - Behaviour Understanding Fisher slide 17 ECVision Summer School: 5 - Behaviour Understanding Fisher slide 18 School of Informatics, University of Edinburgh School of Informatics, University of Edinburgh SEARCH TREE SEARCHING THE TREE POTENTIAL FULL SEARCH TREE 1. DEPTH-FIRST SEARCH DOWN THE LEFT EDGES MATCHED 2. IF ROLE GROUP TYPE DOES NOT MATCH THE ROLES SITUATION, THEN FAIL THIS PATH AND BACKTRACK 3. SUCCESS ONLY IF YOU REACH LEVEL N (EXPLAINS R0 M B * ALL N ROLE GROUPS) R0,R1 4. MULTIPLE SUCCESSES POSSIBLE (EG. RIGHT EDGE IS MB M* BM B* *M *B ** ‘***...*’) ... R0,R1,R2 MBM MB* 5. COUNT NUMBER OF FRAMES (NOT ROLE GROUPS) EVERY POSSIBLE INTERPRETATION TO MATCHED TO ‘*’ ROLE GROUP SEQUENCE R 1 , R 2 , . . . R N 6. SELECT SOLUTION WITH THE SMALLEST COUNT WITH BROWSE CONTEXT MODEL ECVision Summer School: 5 - Behaviour Understanding Fisher slide 19 ECVision Summer School: 5 - Behaviour Understanding Fisher slide 20

School of Informatics, University of Edinburgh School of Informatics, University of Edinburgh RECOGNITION RESULTS EXTENSIONS • ROLE INSTANCES BEFORE PREPROCESSING: 37086 • MISLABELED ROLE INSTANCES BEFORE & AFTER 1. MULTIPLE CONTEXTS: TRY ALL AND SELECT PREPROCESSING I: 4083 & 2843 CONTEXT WITH SMALLEST * FRAME COUNT • MERGED GROUPS OF CONSECUTIVE ROLES BEFORE 2. AMBIGUITY OF ROLE TYPES: MATCH WITH BOTH PREPROCESSING II: 170 (2822 STILL MISLABELLED LABELS ROLES) 3. MODEL FOR TEMPORAL LENGTH OF SITUATIONS • GROUPS JOINED TO NEIGHBOURS IN PREPROCESSING 4. CONTEXT CHANGES OR CONSECUTIVE SCENARIOS II: 6 5. GEOMETRY OF TRAJECTORIES (IGNORED) • ROLE GROUPS AFTER PREPROCESSING II: 164 6. INTERACTIONS (SEE LATER) • FRAMES RELABELLED IN PREPROCESSING II: 90 • FRAMES MATCHED TO WILDCARD: 449 OF 25665 ECVision Summer School: 5 - Behaviour Understanding Fisher slide 21 ECVision Summer School: 5 - Behaviour Understanding Fisher slide 22 School of Informatics, University of Edinburgh School of Informatics, University of Edinburgh RECOGNITION RESULTS II CORRECTLY WHAT WE HAVE LEARNED SCENARIO POSSIBLE RECOGNISED BROWSE 10 7 1. SYMBOLIC BEHAVIOUR RECOGNITION WALK 70 44 INACTIVE 46 29 ALGORITHM DROP DEAD 5 1 2. HOW TO APPLY CROWLEY’S TOTAL 131 81 (62%) BEHAVIOUR MODEL FAILURE CAUSES: MULTIPLE SCENARIOS IN SINGLE TRACKING; TOO SHORT TIME IN SITUATION MORE WORK IN PROGRESS ECVision Summer School: 5 - Behaviour Understanding Fisher slide 23 ECVision Summer School: 5 - Behaviour Understanding Fisher slide 24

School of Informatics, University of Edinburgh School of Informatics, University of Edinburgh EXPECTATION GRAMMARS USE DOMAIN KNOWLEDGE TO EXPLAIN Lecture Problem SEQUENCES DESCRIBE THE RECOGNITION RESULT FOR THE BROWSE EXPRESS IN SYMBOLIC FORM USING CONTEXT ON THIS SEQUENCE OF GRAMMAR ROLE GROUPS { MOVE, BROWSE, USEFUL IN HIGHLY CONSTRAINED ENTER, MOVE } . DRAW THE DOMAINS INTERPRETATION TREE. BASED ON MINNEN, ESSA, STARNER ECVision Summer School: 5 - Behaviour Understanding Fisher slide 25 ECVision Summer School: 5 - Behaviour Understanding Fisher slide 26 School of Informatics, University of Edinburgh School of Informatics, University of Edinburgh TOWER OF HANOI TASK EXAMPLE PROBLEM IMAGE UNDERSTAND WHAT IS HAPPENING IN THIS FRAME GOAL: MOVE WHOLE STACK TO NEW PEG CAN ONLY MOVE 1 RING EACH TIME ONLY SMALLER RINGS ON LARGER ECVision Summer School: 5 - Behaviour Understanding Fisher slide 27 ECVision Summer School: 5 - Behaviour Understanding Fisher slide 28