experiment presentation by nayan singhal motivation human
play

Experiment presentation by Nayan Singhal Motivation Human - PowerPoint PPT Presentation

Situation Recognition: Visual Semantic Role Labelling for Image Understanding Mark Yatskar, Luke Zettlemoyer, Ali Farhadi Experiment presentation by Nayan Singhal Motivation Human understanding of image Verbs in English language.


  1. Situation Recognition: Visual Semantic Role Labelling for Image Understanding Mark Yatskar, Luke Zettlemoyer, Ali Farhadi Experiment presentation by Nayan Singhal

  2. Motivation ● Human understanding of image ● Verbs in English language.

  3. Approach ● CRF with CNN CARRYING AGENT WOMAN 1024 CNN CRF ITEM JAR AGENTPART HEAD PLACE OUTDOOR ● Log linear Loss

  4. How object plays role in image understanding?

  5. neighboring images

  6. Remove Cliff Removing Cliff neighboring images

  7. Remove person neighboring images Removing Man

  8. Remove Sky neighboring images Removing Sky

  9. Image (2) neighboring images

  10. Remove Person neighboring images Removing man

  11. Remove Background neighboring images Removing Sky and Man

  12. Conclusion Each object plays a significant role in image understanding.

  13. Experiment 1) Analyzing Failure Cases 2) Different moods of faces

  14. Expt 1: Analyzing Failure Cases

  15. Object Recognition (1) Imsitu Result

  16. Object Recognition (2) Imsitu Result

  17. Object Recognition (3) Imsitu Result

  18. Object Recognition (4) Imsitu Result

  19. Object Recognition (5) Imsitu Result

  20. Object Recognition (6) Imsitu Result

  21. Why is it happening? Are these images difficult to categorize?

  22. Let’s analyze these with ImageNet

  23. Object Recognition (1) Imsitu Result

  24. ImageNet classification

  25. Object Recognition (2) Imsitu Result

  26. Object Recognition (2) ImageNet classification

  27. Object Recognition (3) Imsitu Result

  28. Object Recognition (3) ImageNet classification

  29. Object Recognition (4) Imsitu Result

  30. Object Recognition (4) ImageNet classification

  31. Object Recognition (5) Imsitu Result

  32. Object Recognition (5) ImageNet classification

  33. Object Recognition (6) Imsitu Result

  34. Object Recognition (6) ImageNet classification

  35. Object Recognition (1) Noun Slot Verb Role Noun Potential Labels A (Verb Role Noun Potential) + B (Labels) Post Processing:

  36. Object Recognition (1) Imsitu Result

  37. Object Recognition (1) VGG Imagenet Verb Potential Verb Role Noun Potential Labels Preprocessing

  38. Future Work ● Add labels in preprocessing.

  39. Exp 2: Different moods ● Laughing ● Smiling ● Frowning ● Grimacing ● Winking ● Squinting ● Shouting ● Puckering Winking Squinting Puckering Smiling Laughing Frowning

  40. Success Case Smiling Laughing Shouting Agent place Agent place Agent place man - man - man - 0.35967 0.35777 0.37531

  41. Success Case Frowning Grimacing Agent place Agent place man - man - 0.24378 0.21052

  42. Failure Case Winking Puckering Agent place Agent place man - woman - 0.20954 0.21052

  43. Test Images (25) ● Conclusion: Detect different moods of faces with slight variation.

  44. Some Interest Categorization

  45. Some Interest Categorization Camouflaging Camouflaging Agent owl Agent frog Hiding Item tree Hiding Item pebble Place outdoors Place -

  46. neighboring images

  47. neighboring images

  48. Thank You

  49. No Agent(2) Watering Shredding Agent Person Agent Person Tool Bucket Tool Shreder Place garden Item paper Place -

Recommend


More recommend