COMP 150: Probabilistic Robotics for Human-Robot Interaction Instructor: Jivko Sinapov www.cs.tufts.edu/~jsinapov
Today: Perception beyond Vision
Announcements
Project Deadlines ● Project Presentations: Tuesday May 5 th 3:30- 6:30 pm ● Final Report + Deliverables: May 10 ● Deliverables: – Presentation slides + videos – Final Report (PDF) – Source code (link to github repositories + README)
ZCam (RGB+D) Microphones in the head Logitech Webcam Torque sensors in the joints 3-axis accelerometer Sinapov, J., Schenck, C., Staley, K., Sukhoy, V., and Stoytchev, A. (2014) Grounding Semantic Categories in Behavioral Interactions: Experiments with 100 Objects Robotics and Autonomous Systems, Vol. 62, No. 5, pp. 632-645, May 2014.
100 objects from 20 categories
Exploratory Behaviors grasp lift hold shake drop tap poke push press
Coupling Action and Perception Action: poke … … … Perception: optical flow … … … Time
Sensorimotor Contexts audio haptics (joint proprioception Optical Color SURF (DFT) torques) (finger pos.) flow look grasp lift hold shake drop tap poke push press
Overview Interaction with Object Category Estimates … Sensorimotor Feature Category Extraction Recognition Model
Context-specific Category Recognition M poke-audio Observation from Recognition model Distribution over poke-audio context for poke-audio category labels context
Recognition Rates with SVM Audio Proprioception Color Optical Flow SURF All look 58.8 58.9 67.7 grasp 45.7 38.7 12.2 57.1 65.2 lift 48.1 63.7 5.0 65.9 79.0 hold 30.2 43.9 5.0 58.1 67.0 shake 49.3 57.7 32.8 75.6 76.8 drop 47.9 34.9 17.2 57.9 71.0 tap 63.3 50.7 26.0 77.3 82.4 push 72.8 69.6 26.4 76.8 88.8 poke 65.9 63.9 17.8 74.7 85.4 press 62.7 69.7 32.4 69.7 77.4
Combining Model Outputs . . . . . . . . M look-color M tap-audio M lift-SURF M press-prop. Weighted Combination
Combining Multiple Behaviors and Modalities
Deep Models for Non-Visual Perception Tatiya, G., and Sinapov, J. (2019) Deep Multi-Sensory Object Category Recognition Using Interactive Behavioral Exploration 2019 IEEE International Conference on Robotics and Automation (ICRA), Montreal, Canada, May 20-24, 2019.
Haptics
Sensorimotor Word Embeddings Sinapov, J., Schenck, C., and Stoytchev, A. (2014). Learning Relational Object Categories Using Behavioral Exploration and Multimodal Perception In the Proceedings of the 2014 IEEE International Conference on Robotics and Automation (ICRA)
Sensorimotor Word Embeddings Thomason, J., Sinapov, J., Stone, P., and Mooney, R. (2018) Guiding Exploratory Behaviors for Multi-Modal Grounding of Linguistic Descriptions In proceedings of the 32nd Conference of the Association for the Advancement of Artificial Intelligence (AAAI)
Integration into Service Robotics Platform Robot, fetch me the green empty bottle
Thomason, J., Padmakumar, A., Sinapov, J., Walker, N., Jiang, Y., Yedidsion, H., Hart, J., Stone, P. and Mooney, R.J. (2020) Jointly improving parsing and perception for natural language commands through human-robot dialog Journal of Artificial Intelligence Research 67 (2020)
How do we extend this approach beyond individual words applied to individual objects?
Language Related to Space and Locations
Example: Offices
Labs
Kitchen
Lobby
Adjectives: clean vs messy
Crowded vs Empty
Computer Vision Solution: Scene Recognition [https://people.csail.mit.edu/bzhou/image/cover_places2.png]
Breakout Activity ● Brainstorm: how can multimodal perception be used to ground language related to locations and places? ● What are some modalities / sources of information beyond visual input that may come useful? ● What are some nouns / adjectives related to locations and places that may be difficult to ground using vision alone? ● Take notes! After 15 minutes we’ll reconvene and share what we found
Student-led Paper Presentation
Recommend
More recommend
