bv lc
play

BV LC Jacob Andreas, Marcus Rohrbach, Trevor Darrell, Dan Klein - PowerPoint PPT Presentation

Sep 26, 2023 •627 likes •1.18k views

Neural Module Networks BV LC Jacob Andreas, Marcus Rohrbach, Trevor Darrell, Dan Klein Visual ques>on answering What color is yellow the neck/e? 2 Visual ques>on answering Is there a red yes shape above a circle? 3

  1. Neural Module Networks BV LC Jacob Andreas, Marcus Rohrbach, 
 Trevor Darrell, Dan Klein

  2. Visual ques>on answering What color is 
 yellow the neck/e? 2

  3. Visual ques>on answering Is there a red yes shape above 
 a circle? 3

  4. Neural module networks Is there a red shape 
 above a circle? ↦ red ↦ true exists ↦ above 4

  5. Neural module networks Is there a red shape 
 above a circle? exists and red above ↦ circle red ↦ true exists ↦ above 5

  6. Neural module networks yes Is there a red shape 
 above a circle? exists and red above ↦ circle red ↦ true exists ↦ above 6

  7. Nearest neighbors Structured neural models • [Socher et al. 2011, BoIou et al. 1997, Mnih et al. 2014] Probabilis7c formal seman7cs / predicate learning • [Beltagy et al. 2013, Lewis & Steedman 2013, 
 Malinowski & Fritz 2014] 7

  8. Represen>ng meaning Is there a red shape above a circle? 8

  9. Represen>ng meaning Is there a red shape above a circle? 9

  10. AIen>ons encode meaning Is there a red shape above a circle? 10

  11. AIen>on transforma>ons encode meaning Is there a red shape above a circle? 11

  12. Set transforma>ons encode meaning Is there a red shape above a circle? 12

  13. Sentence meanings are computa>ons Is there a red shape above a circle? exists and red above circle 13

  14. Sentence meanings are computa>ons Is there a red shape above a circle? exists and red above circle 14

  15. Composi>ons of vector func>ons are neural nets 15

  16. Outline yes Is there a red shape 
 above a circle? exists and red above ↦ circle red ↦ true exists ↦ above 16

  17. Outline yes Is there a red shape 
 above a circle? exists and red above ↦ circle red ↦ true exists ↦ above 17

  18. Outline yes Is there a red shape 
 above a circle? exists and red above ↦ circle red ↦ true exists ↦ above 18

  19. The [find] module red 19

  20. The [find] module necktie 20

  21. The [find] module red 21

  22. The [find] module red 22

  23. The [find] module red red 23

  24. The [find] module 0.9 red red 24

  25. The [find] module 0.9 red red 25

  26. The [find] module 0.1 red red 26

  27. The [describe] module red color 27

  28. The [describe] module neck/e what 28

  29. The [describe] module red color 29

  30. The [describe] module red . . . color 30

  31. The [describe] module red . . . color 31

  32. Outline yes Is there a red shape 
 above a circle? exists and red above ↦ circle red ↦ true exists ↦ above 32

  33. Where do layouts come from? Is there a red shape above a circle? be and shape there shape red above any red above circle circle a [Reddy et al. 2016] 33

  34. Learning yes blue exists color and right_of red above circle circle Is there a red shape above a circle? What color is the shape right of a circle? 34

  35. Learning yes blue Is there a red shape above a circle? What color is the shape right of a circle? 35

  36. Parameter tying yes blue circle circle Is there a red shape above a circle? What color is the shape right of a circle? 36

  37. Parameter tying yes blue circle circle Is there a red shape above a circle? What color is the shape right of a circle? 37

  38. Extreme parameter tying exists exists color shape and and right_of right_of red above above red circle square circle left_of circle 38

  39. Learning with fixed layouts is easy! Σ p( | ; W ) arg max , yes W (where every root module outputs a distribu>on over answers and W is the set of all module parameters) 39

  40. Learning module behaviors Module specializa>on is driven en>rely 
 right_of by context! square 40

  41. Experiments 41

  42. Experiments: VQA dataset What color 
 yellow is the neck/e? What is in the tag sheep’s ear? [Antol et al. 2015] 42

  43. Experiments: VQA dataset 60.00 59.4 58.9 56.67 57.4 55.9 53.33 Noh Yang Zhou Ours (2015) (2015) (2015) 50.00 43

  44. Experiments: VQA dataset 60.00 59.4 58.9 56.67 57.4 55.9 53.33 Noh Yang Zhou Ours (2015) (2015) (2015) 50.00 44

  45. Experiments: VQA dataset What color is 
 color white she wearing? and wear 45

  46. Experiments: VQA Dataset What color is 
 color white she wearing? and wear 46

  47. Experiments: VQA Dataset what What is in the tag and and sheep’s ear? sheep ear 47

  48. Experiments: VQA Dataset what What is in the tag and and sheep’s ear? sheep ear 48

  49. Experiments: VQA Dataset what What is in the tag and and sheep’s ear? sheep ear 49

  50. Experiments: SHAPES dataset Is there a red yes shape above 
 a circle? Is a green shape no above le< of a red shape? c.f. [Xu and Saenko, 2016] 50

  51. Experiments: SHAPES dataset 100.00 87.50 90.6 75.00 76.5 62.50 65.3 *Zhou *Yang Ours 50.00 51

  52. Experiments: SHAPES dataset find[red] attend[red] yes combine[and] exists combine[and] measure[is] relate[above] find[circle] attend[circle] re-attend[above] 52

  53. Neural module networks yes Is there a red shape 
 above a circle? exists and red above ↦ circle red ↦ true exists ↦ above 53

  54. Neural module networks Linguis7c structure dynamically generates model structure Combines advantages of: exists and • Representa>on learning (like a neural net) red above circle • Composi>onality (like a seman>c parser) 54

  55. thank you Download our code at hIp://github.com/jacobandreas/nmn2

Recommend

More recommend