Hashing for Multi-Label Image Retrieval Fang Zhao, et al. CVPR - PowerPoint PPT Presentation

Deep Semantic Ranking Based Hashing for Multi-Label Image Retrieval Fang Zhao, et al. CVPR 2015 Presenter: MinKu Kang 1

Previous Presentation – Presented by Youngki Kwon 2

Introduction – Ranking Based Image Retrieval Similarity based on # common labels 3

Previous Work - Metric Learning Pairwise Similarity Dissimilar Similar Assumed each image contains a single representative label 4

Previous Work - Metric Learning Assumed each image contains a single representative label 5

Previous Work – Triplet Network . CNN 𝐺(𝐽) . . 𝐽 𝐺(𝐽 + ) . CNN . . 𝐽 + Weights are shared. Triplet Ranking Loss . CNN 𝐺(𝐽 − ) . . 𝐽 − Weights are shared. Assumed each image contains a single representative label 6

Multi-label based Ranking more-similar less-similar Count the number of common labels 7

Ranking Score 𝑠 2 = 2 𝑠 1 = 3 𝐵𝑜𝑑ℎ𝑝𝑠 𝑢𝑠𝑓𝑓, 𝑡𝑙𝑧 𝑢𝑠𝑓𝑓, 𝑡𝑣𝑜, 𝑡𝑙𝑧 𝑢𝑠𝑓𝑓, 𝑡𝑣𝑜, 𝑡𝑙𝑧 dissimilar Similar 8

Triplet Loss Function 𝑠 2 = 2 𝑠 1 = 3 𝐵𝑜𝑑ℎ𝑝𝑠 𝑢𝑠𝑓𝑓, 𝑡𝑙𝑧 𝑢𝑠𝑓𝑓, 𝑡𝑣𝑜, 𝑡𝑙𝑧 𝑢𝑠𝑓𝑓, 𝑡𝑣𝑜, 𝑡𝑙𝑧 dissimilar Similar 9

Constructing Triplets Loss for an anchor = Possible Triplets 𝑠 2 = 2 𝑠 3 = 0 𝒙𝒊𝒇𝒐 𝒋 = 𝟐 𝑠 1 = 3 𝑠 1 = 3 𝑢𝑠𝑓𝑓, 𝑡𝑙𝑧 𝑐𝑣𝑗𝑚𝑒𝑗𝑜𝑕, 𝑑𝑏𝑠 𝑢𝑠𝑓𝑓, 𝑡𝑣𝑜, 𝑡𝑙𝑧 𝑢𝑠𝑓𝑓, 𝑡𝑣𝑜, 𝑡𝑙𝑧 𝒓 𝒚 𝒋 𝒚 𝒌 𝒚 𝒌 𝒙𝒊𝒇𝒐 𝒋 = 𝟑 𝑠 3 = 0 𝑠 2 = 2 𝑠 1 = 3 𝑢𝑠𝑓𝑓, 𝑡𝑣𝑜, 𝑡𝑙𝑧 𝑢𝑠𝑓𝑓, 𝑡𝑙𝑧 𝑐𝑣𝑗𝑚𝑒𝑗𝑜𝑕, 𝑑𝑏𝑠 10

Final Loss Function Loss for an anchor Loss for all anchors 11

Final Loss Function – Regularizers … : Encourages each bit averaged over the training data to be mean-zero : Penalized large weights 12

Skipping Layer Skipping Layer Bypassing Connection Utilize diverse feature information biased toward visual appearance 13

Additional Relaxations discontinuous Relaxation smooth, differentiable 14

Additional Relaxations discontinuous Relaxation smooth, differentiable But, the sigmoid function had a bad influence on the convergence of the network Many Deep Learning Libraries support automatic, symbolic differentiations 15

Additional Modification on Loss Function 𝑠 1 = 3 𝑠 2 = 2 Ranking Discrepancy 𝑠 1 = 3 𝑠 3 = 1 Higher weight 16

Experiments – Multi-Labeled Dataset MIRFLICKR-25K : Multi-label(24) images from social photography website NUS-WIDE : Multi-label(81) images 17

Experiments - Measure For top-p retrieved images Average Ranking 18

Experimental Results MIRFLICKR-25K NUS-WIDE 19

Experimental Results – Effect of Skipping Layer / Weighting Scheme MIRFLICKR-25K NUS-WIDE 20

Summary 21

Quiz 1. What is the most appropriate role of the bypassing connection? a) to increase the capacity of the network b) to utilize the diverse feature information c) to make the training procedure efficient. d) to prevent the overfitting training dataset 2. Choose the term which is not included in the final loss function. a) mean-zero relaxation b) sign function => sigmoid function term c) large-weight penalizing term d) L1 regularization term for the hash codes 22

References • Deep Semantic Ranking Based Hashing for Multi-Label Image Retrieval, Fang Zhao, et al., CVPR 2015 23