Local Feature Extraction and Learning for Computer Vision Part 3: - PowerPoint PPT Presentation

IEEE CVPR 2017 Tutorial on Local Feature Extraction and Learning for Computer Vision Part 3: Binary Feature Learning for Visual Recognition and Search Jiwen Lu Department of Automation, Tsinghua University, China http://ivg.au.tsinghua.edu.cn/Jiwen_Lu/

Visual Recognition 2

Visual Search 3

Towards Efficient Descriptors (Binary) •Handcrafted Descriptors • BRIEF, BRISK, FREAK, FRIF •Learned Descriptors • Learning to threshold and Select • Traditional Learning • Deep Learning 4

Towards Efficient Descriptors (Binary) [ ] = τ τ ∈ {0,1} n  f ( ) P ( ; P x y , ), , ( ; P x , y ) n 1 1 n n >  1, ( ) P x P y ( ) τ =  ( ; , ) P x y ≤  5 0, ( ) P x P y ( )

Learning Local Binary Features for Visual Recognition • A conventional visual recognition system • Offline: training model, gallery feature extraction, storage • Online: probe feature extraction, matching 6

Learning Local Binary Features for Visual Recognition • Binary descriptors present high storage efficiency and matching speed • Efficient storage • Real-valued descriptors -> Binary codes • Fast matching • Euclidean distance -> Hamming distance 7

Learning Local Binary Features for Visual Recognition • Local Binary Feature Descriptor: LBP [Ahonen et al, ECCV 2004] 8

Learning Local Binary Features for Visual Recognition Bin distribution of LBP Bin distributions in the LBP histogram in the FERET training set. 9

Learning Local Binary Features for Visual Recognition [1] Jiwen Lu , Venice Erin Liong, Xiuzhuang Zhou, and Jie Zhou, Learning compact binary face descriptor for face recognition, IEEE Trans. on Pattern Analysis and Machine Intelligence , vol. 37, no. 10, pp. 10 2041-2056, 2015.

Learning Local Binary Features for Visual Recognition Objective • First term: redundancy removing • Second term: energy preserving • Third term: balanced bin 11

Learning Local Binary Features for Visual Recognition Bin distribution of CBFD Bin distributions in the CBFD histogram in the FERET training set. 12

Learning Local Binary Features for Visual Recognition 13 image-unrestricted setting image-restricted setting

Learning Local Binary Features for Visual Recognition Two-step procedure in LBP 14

Learning Local Binary Features for Visual Recognition SLBFLE 15 [2] Jiwen Lu , Venice Erin Liong, and Jie Zhou, Simultaneous local binary feature learning and encoding for face recognition ICCV pp 3721 3729 2015

Learning Local Binary Features for Visual Recognition Results on LFW 16

Learning Local Binary Features for Visual Recognition CS-LBFL [3] Jiwen Lu , Venice Erin Liong, and Jie Zhou, Cost-sensitive local binary feature learning for facial age estimation, IEEE Transactions on Image Processing, vol. 24, no. 12, pp. 5356-5368, 2015. 17

Learning Local Binary Features for Visual Recognition CS-LBFL • First term: large margin • Second term: cost-sensitive Cost function 18

Learning Local Binary Features for Visual Recognition Comparisons with state-of-the-arts 19

Learning Local Binary Features for Visual Recognition • Motivation – Exploit contextual information of binary codes as strong prior knowledge to enhance the robustness [4] Yueqi Duan, Jiwen Lu , Jianjiang Feng, and Jie Zhou, Context-aware local binary feature learning for face recognition, IEEE Trans. on Pattern Analysis and Machine Intelligence , 2017, accepted. 20

Learning Local Binary Features for Visual Recognition • LFW 21

Learning Local Binary Features for Visual Recognition • Rotation-invariance [5] Yueqi Duan, Jiwen Lu , Jianjiang Feng, and Jie Zhou, Learning rotation-invariant local binary descriptor, IEEE Trans. on Image Processing, vol. 26, no. 8, pp. 3636-3651, 2017. 22

Learning Local Binary Features for Visual Recognition • RBP: Describe the circular changing tendency of a local patch 23

Learning Local Binary Features for Visual Recognition • Outex-TC12 24

Learning Local Binary Features for Visual Recognition [6] Kevin Lin, Jiwen Lu , Chu-Song Chen, and Jie Zhou, Learning compact binary descriptors with 25 unsupervised deep neural networks, CVPR , pp. 1183-1192, 2016.

Learning Local Binary Features for Visual Recognition 26

Learning Local Binary Features for Visual Recognition 27

Learning Local Binary Features for Visual Recognition [7] Yueqi Duan, Jiwen Lu , Ziwei Wang, Jianjiang Feng, and Jie Zhou, Learning deep binary descriptor with multi-quantization, CVPR , 2017, accepted. 28

Learning Local Binary Features for Visual Recognition • Sign function ignores data distributions 29

Learning Local Binary Features for Visual Recognition • Train K-autoencoders (KAEs) with an iterative two- step procedure 30

Learning Local Binary Features for Visual Recognition • Brown 31

Learning Binary Feature for Visual Search • Image and Video search • Find most similar images/videos • Search engine • Collaborative filtering • Product search • Medical search • Person re-identification 32

Learning Binary Feature for Visual Search • Similarity measurement • Hamming distance • Storage • Short binary codes • Encoding strategy • Hashing functions H=[h 1 , h 2 , …, h n ] • Binary code for sample x 1 , B 1 =[h 1 (x 1 ), h 2 (x 1 ), …, h n (x 1 )] 33

Learning Binary Feature for Visual Search • Design of hashing function is crucial for effective search. • Goal: Compact yet discriminative binary codes. Hashing functions Hamming distance 34

Learning Binary Feature for Visual Search [8] Venice Erin Liong, Jiwen Lu, Gang Wang, Pierre Moulin, and Jie Zhou, Deep hashing for compact 35 binary codes learning, CVPR , pp. 2475-2483, 2015.

Learning Binary Feature for Visual Search 36

Learning Binary Feature for Visual Search • Multi-label extension – Re-formulate the between-class and within-class scatter matrix of SDH for multi-label samples [9] Jiwen Lu , Venice Erin Liong, and Jie Zhou. Deep hashing for scalable image search, IEEE Trans. on Image Processing , vol. 26, no. 5, pp. 2352-2367, 2017. 37 37

Learning Binary Feature for Visual Search NDCG ACG 38

Learning Binary Feature for Visual Search • Motivation – Exploit the nonlinear relationship of samples with nonlinear hashing functions – Solving the discrete optimization problem to eliminate the quantization error accumulation [10] Zhixiang Chen, Jiwen Lu , Jianjiang Feng, and Jie Zhou. Nonlinear discrete hashing , IEEE Trans. on Multimedia, vol. 19, no. 1, pp. 123-135, 2017. 39

Learning Binary Feature for Visual Search 40

Learning Binary Feature for Visual Search • Deep Video Hashing Extract features Exploit both the for each frame Handle entire temporal and video with a deep discriminative learning framework information Image hashing techniques [11] Venice Erin Liong, Jiwen Lu , Yap-Peng Tan, and Jie Zhou. Deep video hashing, IEEE Trans. on Multimedia , vol. 19, no. 6, pp. 1234-1244, 2017. 41

Learning Binary Feature for Visual Search Slow fusion Early fusion Late fusion 42

Learning Binary Feature for Visual Search • Formulation – J1: discriminative learning. Minimize the intra-class variation and maximize the inter-class variation of the binary feature representation. – J2: efficient binary coding with minimizing the quantization loss. 43

Learning Binary Feature for Visual Search • Extracting binary codes from one video 44

Learning Binary Feature for Visual Search 45

Learning Binary Feature for Visual Search [12] Zhixiang Chen, Jiwen Lu , Jianjiang Feng, and Jie Zhou. Nonlinear structural hashing for scalable video search, IEEE Transactions on Circuits and Systems for Video Technology , 2017, accepted. 46

Learning Binary Feature for Visual Search 47

Summary and Future Work • Learning local binary features is very effective for many visual analysis tasks including visual recognition and search tasks. • More efforts are desirable to further improve its real applications, especially on unsupervised hashing and structural hashing. • New criterions are also required to better evaluate the performance of different hashing methods. 48 48

Acknowledge • Prof. Jie Zhou, Dr. Zhixiang Chen, Mr. Yueqi Duan, and Mr. Ziwei Wang from Tsinghua University • Prof. Yap-Peng Tan, Mr. Junlin Hu, and Miss Venice Erin Liong from Nanyang Technological University • Mr. Kevin Lin from University of Washington 49

Thank you! 50 50