Theory Mind from Videos Tao Gao MIT Theory of Mind Can't see - PowerPoint PPT Presentation

Theory Mind from Videos Tao Gao MIT

Theory of Mind Can't see Can't do Don't care R B A M

Theory of Mind 1. Dimensionality 4. Observability 2. Optimality 5. Recursion 3. Physics 6. Hierarchy

Theory of Mind 1. Dimensionality 4. Observability 2. Optimality 5. Recursion 3. Physics 6. Hierarchy

Dimensionality Point Dot Full Body

Don’t-Get-Caught

Don’t-Get-Caught Wolf Succeed Kill after 20s! zone Gao & Scholl (2011, 2013)

Perfect Heat Seeking

Chasing Subtlety …maximum deviation from heat seeking Sheep Sheep Sheep Heat Seeking Heat Seeking Heat Seeking 90° ° 0 3 Wolf Wolf Wolf Chasing Subtlety =0° Small Subtlety Mild Subtlety (Perfect Heat Seeking) (30°) (90°)

Subtlety 90º (Cheating)

90° Subtlety

Dynamic Bayesian Model θ ~ [ 5,30,60,90,120,150,180 ] C : onehot encodingwith dim N s ) 1: N | L 0: t p ( C , θ , L 0: t θ C s )∝ p ( C , θ , L 0: t s ) 1: N ,L 0: t 1: N | L 0: t p ( C , θ | L 0: t 1: N 1: N L t L t − 1 s )∝ ∑ θ p ( C , θ ,L 0: t s ) 1: N L 0: t 1: N | L 0: t p ( C | L 0: t s ) 1: N | C , θ ,L i − 1 1: N ,L i − 1 s s L t − 1 L t p ( L i t s ) = p ( C ) p (θ) ∏ s ) 1: N | C , θ ,L i − 1 1: N ,L i − 1 1: N | L 0: t p ( C , θ , L 0: t p ( L i i = 1

Dimensionality Point Dot Full Body

Baby: I can help What ? Why ? Should ? How ? Warneken & Tomasello (2006) Science

Live Experiment

20%

20%

Theory of Mind 1. Dimensionality 4. Observability 2. Optimality 5. Recursion 3. Physics 6. Hierarchy

Optimality Heuristics Rational Planning

Chasing Subtlety Sheep Heat Seeking Wolf Chasing Subtlety =0° (Perfect Heat Seeking)

Rational planning

Rational Planning

Rational Planning

Theory of Mind 1. Dimensionality 4. Observability 2. Optimality 5. Recursion 3. Physics 6. Hierarchy

Physics Dynamics Kinematics

Kinematic Wolf

Dynamic 2D physics simulation Force control

Dynamics Mordatch I, Todorov E & Popovic, Z (2012)

Theory of Mind 1. Dimensionality 4. Observability 2. Optimality 5. Recursion 3. Physics 6. Hierarchy

Observability Partial Fully Observable Observable

MIT Food Truck Study K:Korean L:Lebanese M: Mexican Baker, Jara-Ettinger, Saxe & Tenenbaum (2017)

Beyond what you can/cannot see ...

Thoughts are unobservable Two things awe me most, the starry sky above me and the moral law within me . -Kant

Theory of Mind 1. Dimensionality 4. Observability 2. Optimality 5. Recursion 3. Physics 6. Hierarchy

Recursion

Recursion Recursion Straight

Recursion … the goal is to hide the goal ...

Nash Equilibrium of Chasing

Theory of Mind 1. Dimensionality 4. Observability 2. Optimality 5. Recursion 3. Physics 6. Hierarchy

Hierarchy Structure Collection

HTN + RRT* Reach Move Grasp Stand Walk

Theory of Mind 1. Dimensionality 4. Observability 2. Optimality 5. Recursion 3. Physics 6. Hierarchy

Collaborators Yibiao Zaho (MIT) Chris Baker (MIT) Paul Vernaza (CMU) Josh Tenenbaum (MIT) Nancy Kanwisher (MIT) Brian Scholl (Yale)

Acknowledgement