Perception of Affordances Perception of Affordances Final Status of - PowerPoint PPT Presentation

Perception of Affordances Perception of Affordances Final Status of Work Final Status of Work Lucas Paletta & Gerald Fritz Lucas Paletta & Gerald Fritz Computational Perception Group Computational Perception Group Institute of Digital Image Processing Institute of Digital Image Processing JOANNEUM RESEARCH Forschungsgesellschaft Forschungsgesellschaft mbH mbH JOANNEUM RESEARCH Graz, Austria Graz, Austria Computational Perception Group (CAPE) MACS YEAR 3 REVIEW MEETING, Sankt Augustin, February 15, 2008

Implementation of Perception Module Implementation of Perception Module Computational Perception Group (CAPE) MACS YEAR 3 REVIEW MEETING, Sankt Augustin, February 15, 2008

Perception Module Perception Module Architecture Architecture Learning Learning of of affordances affordances Execution Execution Control Control Behaviors Behaviors Feature detectors detectors Feature Computational Perception Group (CAPE) MACS YEAR 3 REVIEW MEETING, Sankt Augustin, February 15, 2008

Points of Integration Points of Integration JR Control JR Data Wrapper JR Computational Units JR Sensor Interface Computational Perception Group (CAPE) MACS YEAR 3 REVIEW MEETING, Sankt Augustin, February 15, 2008

Perception Module Perception Module Components Components PERCEPTION MODULE (PM) PERCEPTION MODULE (PM) read read entity trajectory structure 1 ( (EntityTrajectoryId EntityTrajectoryId, , TimeStamp TimeStamp) ) . Entity Frame Monitor Frame Monitor Entity . (EntityFrame EntityFrame) ) ← ( ← (EFM) (EFM) . entity trajectory structure N configEFM configEFM Entity Trajectory Trajectory Cache Cache , Entity (EntityTrajectoryIdList ( EntityTrajectoryIdList, (ETC) (ETC) EntityFrameAttributeList EntityFrameAttributeList) ) configCU configCU Computational Perception Perception Computational Sensor Toolbox (ST) Sensor Toolbox (ST) ( (ComputationalUnitInstId ComputationalUnitInstId, , Toolbox (CPT) Toolbox (CPT) real sensors ParamList ParamList) ) … CU #1 CU #2 CU #N virtual sensors request request Entity Entity Structure Structure (EntityTrajectoryId ( EntityTrajectoryId, , ComputationalUnitId ComputationalUnitId, , ParamList ParamList , , Generator Module Generator Module SampleRate SampleRate, , CacheSize CacheSize) ) (ESGM) (ESGM) ← ( (EntitytTrajectoryId EntitytTrajectoryId, , ← ComputationalUnitInstId ComputationalUnitInstId) ) Computational Perception Group (CAPE) MACS YEAR 3 REVIEW MEETING, Sankt Augustin, February 15, 2008

Perception Module Perception Module Entity Trajectories Entity Trajectories entity trajectory structure ET#1 ET#2 EntityTrajectoryId Computational Computational Perception Perception Toolbox (CPT) Toolbox (CPT) CU #1 CU #1 ComputationalUnitId ComputationalUnitInst InstId Computational Perception Group (CAPE) MACS YEAR 3 REVIEW MEETING, Sankt Augustin, February 15, 2008

Vision based Affordance Cueing Vision based Affordance Cueing Computational Perception Group (CAPE) MACS YEAR 3 REVIEW MEETING, Sankt Augustin, February 15, 2008

Affordance Cueing Affordance Cueing 2D Affordance Cueing 2D Affordance Cueing sit! sit! sit! sit! Computational Perception Group (CAPE) MACS YEAR 3 REVIEW MEETING, Sankt Augustin, February 15, 2008

Affordance Cueing Affordance Cueing Components for Components for Affordance based Recognition Affordance based Recognition CUE CUE BEHAVIOR BEHAVIOR OUTCOME OUTCOME AFFORDANCE CUEING AFFORDANCE CUEING AFFORDANCE RECOGNITION AFFORDANCE RECOGNITION Action-Perception Sensory-Motor Interaction Sensory-Motor Interaction Verify Cycle AFFORDANCE FEATURE AFFORDANCE Final Final Affordance Application RECOGNITION ENTITY ENTITY State State support Affordance Monitoring AFFORDANCE ATTENTION (Top-down) AFFORDANCE ATTENTION (Top-down) Paletta et al., ECVP 2006 Paletta et al., ECVP 2006 Computational Perception Group (CAPE) MACS YEAR 3 REVIEW MEETING, Sankt Augustin, February 15, 2008

Affordance Cueing Affordance Cueing Scenario Scenario Computational Perception Group (CAPE) MACS YEAR 3 REVIEW MEETING, Sankt Augustin, February 15, 2008

Affordance Cueing Affordance Cueing Affordance Feature Matrix Affordance Feature Matrix colour G R M R Y B Bl Gr SIFT category R R C C R R R N ratio L/W L L L L P P P L T/B T T T T B B B N LIFTABLE LIFTABLE Y Y Y Y N N N N Y Y Y Y N N N N NOT LIFTABLE NOT LIFTABLE N N N N Y Y Y Y Y Y Y Y Y Y N N entity attribute attribute configuration configuration entity Computational Perception Group (CAPE) MACS YEAR 3 REVIEW MEETING, Sankt Augustin, February 15, 2008

Affordance Cueing Affordance Cueing Hierarchical Hierarchical configuration object obj. conf. object model Structure Structure scene scene modelling modelling object feature … … …  Affordance perception on object aspect graph manifold all levels levels of representation all object object modelling modelling possible possible  Predefined policies grouping grouping feature histogram geometr. SIFT app. pattern surface graph motion groups previously learned previously learned grouping grouping Gibson‘s view hand-coded hand-coded  Any combination from learning region feature region histogram SIFT app. pattern surface part motion region gather powerful policies gather powerful policies segmentation segmentation feature point colour orientation depth motion … visual Information visual Information Computational Perception Group (CAPE) MACS YEAR 3 REVIEW MEETING, Sankt Augustin, February 15, 2008

Affordance Cueing Affordance Cueing Computational Perception Computational Perception Toolbox Toolbox   Point features: Point features:  Sift/Blob detector  Sift/Blob detector object feature object aspect graph manifold … … …   Region features: Region features:  Color histograms  Color histograms object object modelling modelling  SIFT descriptor  SIFT descriptor   Grouping features: Grouping features: grouping grouping feature histogram geometr. SIFT app. pattern surface graph motion groups   Geometric SIFT Geometric SIFT  DT classification  DT classification grouping grouping   Image enhancement Image enhancement   Motion & motion segmentation Motion & motion segmentation  region  feature region Segmentation Segmentation histogram SIFT app. pattern surface part motion region  Watershed + Energy Merging  Watershed + Energy Merging  Connected Components  Connected Components segmentation segmentation   Grouping Grouping  Histogram  Histogram feature point colour orientation depth motion …  Region Classification  Region Classification  Tracking (KLT)  Tracking (KLT) visual Information visual Information Computational Perception Group (CAPE) MACS YEAR 3 REVIEW MEETING, Sankt Augustin, February 15, 2008

Affordance Cueing Affordance Cueing Affordance Cueing: Affordance Cueing: Implementation Implementation KLT track KLT track color ROI color ROI original original SIFT SIFT class. SIFT class. SIFT class. class. hist hist. . affordance affordance Computational Perception Group (CAPE) MACS YEAR 3 REVIEW MEETING, Sankt Augustin, February 15, 2008

Affordance Cueing Affordance Cueing Real-World Affordance Cueing Real-World Affordance Cueing Computational Perception Group (CAPE) MACS YEAR 3 REVIEW MEETING, Sankt Augustin, February 15, 2008

Affordance Cueing Affordance Cueing Learning the Classification of Learning the Classification of Affordance Cues Affordance Cues top = T Y circular = T unknown N size > 1426 liftable P(A liftable |circ) ≈ 0.00 non liftable Y N P(A nliftable |circ) ≈ 1.00 size < 1410 liftable P(A liftable |rect,T) ≈ 0.99 Y pruning pruning P(A nliftable |rect,T) ≈ 0.01 liftable non liftable Fritz et al., SAB 2006; IROS 2006 Fritz et al., SAB 2006; IROS 2006 Computational Perception Group (CAPE) MACS YEAR 3 REVIEW MEETING, Sankt Augustin, February 15, 2008

Affordance Cueing Affordance Cueing Related Work Related Work  Function based object recognition (Stark & Bowyer 94, Rivlin et al. 95)  GRUFF (Generic Representation Using Form and Function)  Qualitative recognition of 3D parts, spatial relations  Mapping to functional primitives & relations between them  Pre-defined feature and object representations Computational Perception Group (CAPE) MACS YEAR 3 REVIEW MEETING, Sankt Augustin, February 15, 2008

Affordance Cueing Affordance Cueing Frameworks for Recognition Frameworks for Recognition visual visual information information visual visual information information & & function function reconstructive (Marr) function based (Stark/Bowyer) appearance based (Poggio) affordance affordance task based based based purposive (Ballard) selected visual visual information information selected Computational Perception Group (CAPE) MACS YEAR 3 REVIEW MEETING, Sankt Augustin, February 15, 2008

2D-3D Information Fusion 2D-3D Information Fusion for for Multi-Sensor Affordance Cueing Multi-Sensor Affordance Cueing Computational Perception Group (CAPE) MACS YEAR 3 REVIEW MEETING, Sankt Augustin, February 15, 2008