Deep Image Description Rui-Wei Zhao rw.du.zhao@gmail.com 1 - PowerPoint PPT Presentation

Deep Image Description Rui-Wei Zhao rw.du.zhao@gmail.com 1

Outline • Generating descriptions for the whole images {Vinyals2014, Karpathy2014} • Generating descriptions for the regional images {Karpathy2014} � � � 2

Generating descriptions for the whole images 3

Predictive Model f ( s | v ; Θ ) = p ( s 1 | v, s 0 ) p ( s 2 | v , s 0 , s 1 ) ··· p ( s T | v , s 0 , …, s T -1 ) 4

hat RNN W oh straw … W hx W hh straw W hi y START CNN θ c h CNN x 5

hat RNN pros W oh straw … W hx W hh straw W hi y START CNN θ c h 1. Visual and semantic features CNN encoded into a common space. 2. In memory contains image content x and predicted words. 6

hat RNN-cons W oh straw … W hx W hh straw W hi y START CNN θ c h CNN 1. Numerical risks (GD) in training. 2. Predict a word using full memory? x 7

LSTM-RNN y g o o f g f c y g i i h g x x x 8

LSTM-RNN y g o o f g f c g i i g x gate x 9

• i t = g i , t ⌾ g x,t � LSTM-RNN • f t = g f , t ⌾ c t -1 � y • c t = i t + f t � σ ( W gox x t + W goo o t-1 ) • o t = g o , t ⌾ c t g o o f g f c σ ( W gix x t + W gio o t-1 ) σ ( W gfx x t + W gfo o t-1 ) g i i g x tanh( W ix x t + W io o t -1 ) gate x 10

Toy Experiment • Training set (407) • dog & frisbee: 59 • man & ride: 324 • kiss: 24 11

a dog jumps to catch a frisbee . 1674612291_7154c5ab61.jpg 12

y g o h o y g o o g f h f c f g f c g i i g i h i g x x tanh x, h 0 o IMAGE 13

a dog jumps to catch a frisbee . y y a g o g o h o h o g f g f c h f c h f g i g i h i h i tanh tanh x, h 0 x, h 0 o o IMAGE the 0 1 14

a dog jumps to catch a frisbee . y y dog jumps g o g o h o h o g f g f c h f c h f g i g i h i h i tanh tanh a dog a x, h 0 x, h 0 o o a dog 2 3 15

a dog jumps to catch a frisbee . y y to catch g o g o h o h o g f g f c h f c h f g i g i h i h i tanh tanh a dog jumps a dog jumps to x, h 0 x, h 0 o o jumps to 4 5 16

a dog jumps to catch a frisbee . y y a frisbee g o g o h o h o g f g f c h f c h f g i g i h i h i tanh tanh a dog jumps to catch a dog jumps to catch a x, h 0 x, h 0 o o a catch 6 7 17

a dog jumps to catch a frisbee . y . g o h o Inherit previous memory g f c h f Acknowledge previous word Update current memory Predict next word g i h i Until all memory fades out tanh a dog jumps to catch a frisbee x, h 0 o frisbee 8 18

a dog jumps a dog jumps to catch a frisbee . to catch a frisbee . y y a a g o g o h o h o g f g f c h f c h f g i g i h i h i tanh tanh a dog jumps to catch x, h 0 x, h 0 o o the catch 1 6 19

a dog jumps a dog jumps to catch a frisbee . to catch a frisbee . y y dog frisbee g o g o h o h o g f g f c h f c h f g i g i h i h i tanh tanh a dog jumps to catch a a x, h 0 x, h 0 o o a a 2 7 20

a dog jumps to a dog is jumping to catch a frisbee . catch a frisbee . 1674612291_7154c5ab61.jpg 2945036454_280fa5b29f.jpg 21

a dog jumps to a dog is jumping to catch a frisbee . catch a frisbee . y y g o g o h o h o g f g f c h f c h f g i h i g i h i tanh tanh x, h 0 x, h 0 o o IMAGE IMAGE 0 0 22

a dog jumps to a dog is jumping to catch a frisbee . catch a frisbee . y y a a g o g o h o h o g f g f c h f c h f g i h i g i h i tanh tanh x, h 0 x, h 0 o o the the 1 1 23

a dog jumps to a dog is jumping to catch a frisbee . catch a frisbee . y y dog dog g o g o h o h o g f g f c h f c h f g i h i g i h i tanh tanh a a x, h 0 x, h 0 o o a a 2 2 24

a dog jumps to a dog is jumping to catch a frisbee . catch a frisbee . y y jumps is g o g o h o h o g f g f c h f h f c g i g i h i h i tanh tanh a dog a dog x, h 0 x, h 0 o o dog dog 3 3 25

a dog jumps to a dog is jumping to catch a frisbee . catch a frisbee . y y jumps jumping g o g o h o h o g f g f c h f h f c g i h i g i h i tanh tanh a dog a dog is x, h 0 x, h 0 o o dog is 3 4 26

a dog jumps to a dog is jumping to catch a frisbee . catch a frisbee . y y to to g o g o h o h o g f g f c h f c h f g i g i h i h i tanh tanh a dog jumps a dog is jumping x, h 0 x, h 0 o o jumps jumping 4 5 27

a dog jumps to a dog is jumping to catch a frisbee . catch a frisbee . y y catch catch g o g o h o h o g f g f c h f c h f g i h i g i h i tanh tanh a dog jumps to a dog is jumping to x, h 0 x, h 0 o o to to 5 6 28

a dog jumps to a dog is jumping to catch a frisbee . catch a frisbee . y y a a g o g o h o h o g f g f c h f c h f g i h i g i h i tanh tanh a dog jumps to catch a dog is jumping to catch x, h 0 x, h 0 o o catch catch 6 7 29

a dog jumps to a dog is jumping to catch a frisbee . catch a frisbee . y y frisbee frisbee g o g o h o h o g f g f c h f c h f g i h i g i h i tanh tanh a dog jumps to catch a a dog is jumping to catch a x, h 0 x, h 0 o o a a 7 8 30

a dog jumps to a dog is jumping to catch a frisbee . catch a frisbee . y y . . g o g o h o h o g f g f c h f c h f g i h i g i h i tanh tanh a dog jumps to catch a frisbee a dog is jumping to catch a fris x, h 0 x, h 0 o o frisbee frisbee 8 9 31

a black dog is jumping to a dog is jumping to catch a frisbee . catch a frisbee . 1626754053_81126b67b6.jpg 2945036454_280fa5b29f.jpg 32

a black dog is jumping to a dog is jumping to catch a frisbee . catch a frisbee . y y g o g o h o h o g f g f c h f c h f g i g i h i h i tanh tanh x, h 0 x, h 0 o o IMAGE IMAGE 0 0 33

a black dog is jumping to a dog is jumping to catch a frisbee . catch a frisbee . y y a a g o g o h o h o g f g f c h f c h f g i g i h i h i tanh tanh x, h 0 x, h 0 o o the the 1 1 34

a black dog is jumping to a dog is jumping to catch a frisbee . catch a frisbee . y y black dog g o g o h o h o g f g f c h f c h f g i g i h i h i tanh tanh a a x, h 0 x, h 0 o o a a 2 2 35

a black dog is jumping to a dog is jumping to catch a frisbee . catch a frisbee . y y dog dog g o g o h o h o g f g f c h f c h f g i g i h i h i tanh tanh a black a x, h 0 x, h 0 o o a black 3 2 36

a black dog is jumping to a dog is jumping to catch a frisbee . catch a frisbee . y y is is g o g o h o h o g f g f h f h f c c g i g i h i h i tanh tanh a black dog a dog x, h 0 x, h 0 o o dog dog 4 3 37

a black dog is jumping to a dog is jumping to catch a frisbee . catch a frisbee . y y jumping jumping g o g o h o h o g f g f h f h f c c g i g i h i h i tanh tanh a black dog is a dog is x, h 0 x, h 0 o o is is 5 4 38

a black dog is jumping to a dog is jumping to catch a frisbee . catch a frisbee . y y to to g o g o h o h o g f g f c h f c h f g i g i h i h i tanh tanh a black dog is jumping a dog is jumping x, h 0 x, h 0 o o jumping jumping 6 5 39

a black dog is jumping to a dog is jumping to catch a frisbee . catch a frisbee . y y catch catch g o g o h o h o g f g f c h f c h f g i g i h i h i tanh tanh a black dog is jumping to a dog is jumping to x, h 0 x, h 0 o o to to 7 6 40

a black dog is jumping to a dog is jumping to catch a frisbee . catch a frisbee . y y a a g o g o h o h o g f g f c h f c h f g i g i h i h i tanh tanh a black dog is jumping to catc a dog is jumping to catch x, h 0 x, h 0 o o catch catch 8 7 41

a black dog is jumping to a dog is jumping to catch a frisbee . catch a frisbee . y y frisbee frisbee g o g o h o h o g f g f c h f c h f g i g i h i h i tanh tanh a black dog is jumping to catch a dog is jumping to catch a x, h 0 x, h 0 o o a a 9 8 42

a black dog is jumping to a dog is jumping to catch a frisbee . catch a frisbee . y y . . g o g o h o h o g f g f c h f c h f g i g i h i h i tanh tanh a black dog is jumping to catch a f a dog is jumping to catch a fris x, h 0 x, h 0 o o frisbee frisbee 10 9 43

Why all dogs end with “frisbee”? Count last word in training sentences with “dog” and “frisbee”: 86 frisbee 6 yard 4 it 2 other 30 mouth 6 disc 4 ground 2 mouths 15 snow 6 air 4 fence 2 man 15 grass 5 watches 4 beach 2 legs 11 field 5 midair 3 road 2 hand 11 dog 5 background 3 object 2 dogs 8 toy 4 watch 3 boat 1 underfoot 7 water 4 park 3 ball 1 … 44

a man in a blue shirt is a man in a blue shirt is riding a bike on a dirt track . riding a bike on a ramp . 2891617125_f939f604c7.jpg 3640422448_a0f42e4559.jpg 45

a man in a blue shirt is a man in a blue shirt is riding a bike on a dirt track . riding a bike on a ramp . y y g o h o g o h o g f c h f g f c h f g i h i g i h i tanh tanh x, h 0 o x, h 0 o IMAGE IMAGE 0 0 46

a man in a blue shirt is a man in a blue shirt is riding a bike on a dirt track . riding a bike on a ramp . y y a a g o h o g o h o g f c h f g f c h f g i h i g i h i tanh tanh x, h 0 o x, h 0 o the the 0 0 47