School of Informatics, University of Edinburgh School of Informatics, University of Edinburgh SHORT TERM ACTION RECOGNITION PROBLEM ACTION PRIMITIVES, NOT SEQUENCES, NEAR FIELD (BODIES 300 PIXELS HIGH) EG. A HAND WAVE → USE GEOMETRIC MODEL TEMPORALLY LOCAL/SHORT-TERM FAR FIELD (BODIES 3 PIXELS HIGH) IMAGE ANALYSIS/INSTANTANEOUS → USE TRACKING APPEARANCE BASED/VIEWPOINT HERE: MEDIUM FIELD (BODIES 30 SPECIFIC PIXELS HIGH) EFROS, BERG, MORI & MALIK AERFAI 2006 Summer School: 4 - Short Term Action Recognition Fisher slide 1 AERFAI 2006 Summer School: 4 - Short Term Action Recognition Fisher slide 2 School of Informatics, University of Edinburgh School of Informatics, University of Edinburgh GOAL SELECT MATCHING ACTIVITIES FROM DATABASE OF VIDEO CLIPS AERFAI 2006 Summer School: 4 - Short Term Action Recognition Fisher slide 3 AERFAI 2006 Summer School: 4 - Short Term Action Recognition Fisher slide 4
School of Informatics, University of Edinburgh School of Informatics, University of Edinburgh KEY CONCEPT PATTERN OF STABILIZED OPTICAL FLOW PROCESS OVERVIEW AERFAI 2006 Summer School: 4 - Short Term Action Recognition Fisher slide 5 AERFAI 2006 Summer School: 4 - Short Term Action Recognition Fisher slide 6 School of Informatics, University of Edinburgh School of Informatics, University of Edinburgh OPTICAL FLOW DESCRIPTORS 1 GOAL: AGGREGATE SPATIAL PATTERN OF NOISY RELATIVE O.F. IMAGE STABILIZATION O.F. IMAGE = [ ... ( u i , v i ) ... ] THRESHOLD TEMPORAL DIFFERENCE FOR REGIONS OF INTEREST NORMALIZED CROSS-CORRELATION INSIDE ROI FOR LOCALIZATION AERFAI 2006 Summer School: 4 - Short Term Action Recognition Fisher slide 7 AERFAI 2006 Summer School: 4 - Short Term Action Recognition Fisher slide 8
School of Informatics, University of Edinburgh School of Informatics, University of Edinburgh OPTICAL FLOW DESCRIPTORS 2 U OPTICAL FLOW DESCRIPTORS 3 NOISY SO SMOOTH, BUT SMOOTHING CANCELS +/- ASPECTS SOLUTION: SPLIT +/- COMPONENTS V f ( x ) = x IF x ≥ 0 ELSE x = 0 ( u i , v i ) → ( f ( u i ) , f ( − u i ) , f ( v i ) , f ( − v i )) u, v COMPONENTS AERFAI 2006 Summer School: 4 - Short Term Action Recognition Fisher slide 9 AERFAI 2006 Summer School: 4 - Short Term Action Recognition Fisher slide 10 School of Informatics, University of Edinburgh School of Informatics, University of Edinburgh OPTICAL FLOW DESCRIPTORS 4 CONVOLVE WITH GAUSSIAN TO SMOOTH U− U+ U− U+ V− V+ V− V+ DESCRIPTOR: SEQUENCE (50 FRAMES) OF SMOOTHED O.F. WINDOWS AERFAI 2006 Summer School: 4 - Short Term Action Recognition Fisher slide 11 AERFAI 2006 Summer School: 4 - Short Term Action Recognition Fisher slide 12
School of Informatics, University of Edinburgh School of Informatics, University of Edinburgh FRAME i MATCHING DESCRIPTORS I SINGLE FRAME MATCHING 4 a i c ( x, y ) b j m ( i, j ) = � � c ( x, y ) c =1 x,y ∈ I j WHERE FRAME i OF SEQ. a , FRAME j OF SEQ. b c = 1,2,3,4 O.F. COMPONENT ( x, y ) = PIXEL POSITIONS AERFAI 2006 Summer School: 4 - Short Term Action Recognition Fisher slide 13 AERFAI 2006 Summer School: 4 - Short Term Action Recognition Fisher slide 14 School of Informatics, University of Edinburgh School of Informatics, University of Edinburgh MATCHING DESCRIPTORS II TIME WINDOW OF T=50 FRAMES r =+ T/ 2 s =+ T/ 2 S ( i, j ) = � � K ( r, s ) m ( i + r, j + s ) r = − T/ 2 s = − T/ 2 WEIGHTED SUM OF NEARBY IN TIME K S ( i, j ) FRAMES NEAR DIAGONAL IS MATCHING FRAMES STRIPES FROM ACTION PERIODICITY AERFAI 2006 Summer School: 4 - Short Term Action Recognition Fisher slide 15 AERFAI 2006 Summer School: 4 - Short Term Action Recognition Fisher slide 16
School of Informatics, University of Edinburgh School of Informatics, University of Edinburgh EXAMPLE MATCHING SEQUENCES CONFUSION MATRICES BALLET TENNIS FOOTBALL AERFAI 2006 Summer School: 4 - Short Term Action Recognition Fisher slide 17 AERFAI 2006 Summer School: 4 - Short Term Action Recognition Fisher slide 18 School of Informatics, University of Edinburgh School of Informatics, University of Edinburgh WHAT WE HAVE LEARNED 1. SHORT TERM ACTION RECOGNITION TECHNIQUE 2. BASED ON STABILIZED OPTICAL Lecture Problem FLOW OF LOCAL MEDIUM SIZED WHY MUST WE STABILIZE THE WINDOWS IMAGE BEFORE COMPUTING THE OPTICAL FLOW? 3. ENCODES TEMPORAL STRUCTURE BETTER 4. BUT: STILL SOMEWHAT VIEWPOINT AND SCALE DEPENDENT AERFAI 2006 Summer School: 4 - Short Term Action Recognition Fisher slide 19 AERFAI 2006 Summer School: 4 - Short Term Action Recognition Fisher slide 20
Recommend
More recommend
