where are they looking
play

Where are they looking? Adria Recasens*, Aditya Khosla*, Carl - PowerPoint PPT Presentation

Aug 21, 2022 •46 likes •366 views

Where are they looking? Adria Recasens*, Aditya Khosla*, Carl Vondrick, Antonio Torralba Presented by: Surbhi Goel Where are they looking? Follow the gaze of the person and identify the object being looked at Demo:

  1. Where are they looking? Adria Recasens*, Aditya Khosla*, Carl Vondrick, Antonio Torralba Presented by: Surbhi Goel

  2. Where are they looking? Follow the gaze of the person and identify the object being looked at

  3. Demo: http://gazefollow.csail.mit.edu/demo.html

  4. Experiments ● Dataset Visualizations ○ Images in the Dataset ○ Head Locations ○ Gaze Locations/Length ● Model Experiments ○ Qualitative Evaluation ○ Visualizing Gaze Mask and Saliency Map ○ Animal Gaze Following ○ Extending to Short Video

  5. Dataset Visualizations

  6. Training Set Images

  7. Training Set Images

  8. Training Set Images

  9. Heatmaps for Head Location Train Test

  10. Heatmaps for Gaze Location Train Test

  11. Heatmaps for Relative Gaze Location Train Test

  12. Histogram for Length of Gaze Train Test

  13. Observations ● Head/Gaze are concentrated for train and scattered for test ● Relative gaze is concentrated for both ● Gaze length relatively short (0.2 peak)

  14. Model Evaluation

  15. Good Cases

  16. Good Cases

  17. Bad Cases Head fully tilted but missed

  18. Bad Cases Face forward but eyes tilted No object of attention

  19. Bad Cases Back facing

  20. Observations ● Handle groups well ● Gaze location is very accurate, head location often not ● Unable to capture eye movement independent of face orientation ● Fails at a lot of back facing cases

  21. Gaze Mask and Saliency Map

  22. Gaze Mask and Saliency Map ● Gaze Mask incorporates the general direction of gaze ● Saliency Map incorporates the salient objects in image ● Element-wise product captures locations that satisfy both

  23. Gaze Mask and Saliency Map Image with Gaze Gaze Mask Saliency Map

  24. Animal Gaze Follow

  25. Animal Gaze Follow

  26. Animal Gaze Follow Works (almost) for even birds

  27. Animal Gaze Follow Works even when more than one salient object

  28. Animal Gaze Follow ● Model generalizes to animals Initialized with ImageNet which has animal data ○ ● Able to learn properties based on orientation of head ● Point of gaze is not always correct

  29. Extension to a Short Video Apply model per frame of video

  30. Extension to a Short Video Head detector often fails, could use temporal context to improve

  31. Conclusions ● Can be confused with mixed orientations and back-facing ● Model generalizes well to animals ● Could be potentially extended to videos ● Could be applied to other domains?

  32. Thank You!

Recommend

More recommend