Multi-View Representation Learning: Algorithms and Applications - PowerPoint PPT Presentation

Multi-View Representation Learning: Algorithms and Applications Changqing Zhang ( 张长 青 ) Tianjin University, China 2019-10-23

O u t l i n e 1. Background ： Multi-View Learning 2. Multi-View Subspace Representation 3. Multi-View Complete Representation 4. Applications 5. Conclusion

Background ： Multi-View Learning Why Multi-View Learning? View 1 View 2 Ground Truth View 3 Synthetic Multi-View Data Video Surveillance Medical Analysis Self-driving Car Multi-View Data in Real World

Background ： Multi-View Learning Why Multi-View Representation Learning? Multi-Modal Medical Data Analysis • Application: Intelligence Medical Diagnosis • Challenge: Multi-modal Integration Medical Data Representation Learning Diagnosis Representation: The Key for Applications!

Background ： Multi-View Learning Why Multi-View Representation Learning? CCA-based Multi-View Representation Learning CCA (1936)-> KCCA (2006)-> DCCA (2013) CCA: Correlation Maximization!

Multi-View Subspace Representation High-order Multi-View Representation Learning Subspace Representation Multiple Subspaces Self-Reconstruction Self-expression-based Subspace Representation

Multi-View Subspace Representation High-order Multi-View Representation Learning Pairwise correlation High-order correlation 1.What is high-order correlation? 2.What is the difference compared to pairwise manner?  (v) (w) (1) (V) corr( , ) corr( ,..., ) X X X X  v w Find the correlation in a global view! [ICCV’15] Changqing Zhang, Huazhu Fu, Si Liu, Guangcan Liu, Xiaochun Cao, Low-Rank Tensor Constrained Multiview Subspace Clustering , ICCV 2015

Multi-View Subspace Representation High-order Multi-View Representation Learning Subspace Representation High-order Correlation Multi-view Features ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... Key observation: Self-representation matrices are aligned: (1) dimensionality and (2) semantic [ICCV’15] Changqing Zhang, Huazhu Fu, Si Liu, Guangcan Liu, Xiaochun Cao, Low-Rank Tensor Constrained Multiview Subspace Clustering , ICCV 2015

Multi-View Subspace Representation High-order Multi-View Representation Learning How to define the rank of a 3-order tensor? Low-rank Unfolding for a 3-order tensor [ICCV’15] Changqing Zhang, Huazhu Fu, Si Liu, Guangcan Liu, Xiaochun Cao, Low-Rank Tensor Constrained Multiview Subspace Clustering , ICCV 2015

Multi-View Subspace Representation High-order Multi-View Representation Learning Modeling high-order correlation is effective! [ICCV’15] Changqing Zhang, Huazhu Fu, Si Liu, Guangcan Liu, Xiaochun Cao, Low-Rank Tensor Constrained Multiview Subspace Clustering , ICCV 2015 [IJCV’18] Yuan Xie, Dacheng Tao, Wensheng Zhang, Yan Liu, Lei Zhang, Yanyun Qu, On Unifying Multi-View Self-Representation for Clustering by Tensor Multi-Rank Minimization, IJCV 2018

Multi-View Subspace Representation Diversity-induced Multi-View Representation Learning Which Group is better? View-1 View-2 View-1 View-2 [CVPR’15] Xiaochun Cao, Changqing Zhang*, Huazhu Fu, Si Liu, Hua Zhang, Diversity-induced Multiview Subspace Clustering, CVPR 2015

Multi-View Subspace Representation Diversity-induced Multi-View Representation Learning Complementarity->Diversity->Independence • Independence maximization for complementarity [1] Complex Correlation [2] Closed-form Solution HSIC = 0.53, pho = 0.81 HSIC = 0.41, pho = 0 HSIC = 0.14, pho = 0 HSIC = 0, pho = 0 HSIC: Hilbert-Schmidt independence criterion [CVPR’15] Xiaochun Cao, Changqing Zhang*, Huazhu Fu, Si Liu, Hua Zhang, Diversity-induced Multiview Subspace Clustering, CVPR 2015

Multi-View Subspace Representation Diversity-induced Multi-View Representation Learning Ensemble learning-like: good & diversity in a better space Better feature space Reconstruction in Latent Space Smooth Term Information Preservation in Latent Space Make the voters diverse Diversity Regularization [CVPR’15] Xiaochun Cao, Changqing Zhang*, Huazhu Fu, Si Liu, Hua Zhang, Diversity-induced Multiview Subspace Clustering, CVPR 2015

Multi-View Subspace Representation Diversity-induced Multi-View Representation Learning Ablation Experiment for Diversity Term [CVPR’15] Xiaochun Cao, Changqing Zhang*, Huazhu Fu, Si Liu, Hua Zhang, Diversity-induced Multiview Subspace Clustering, CVPR 2015

Multi-View Complete Representation Latent Multi-View Subspace Clustering A flexible Typical: way correlation maximization     (v) 2 (v) 2 || (x ) h || || g (h) x || f 2 2 v v v v An intuitive explanation [CVPR’17/Spotlight] Changqing Zhang, Qinghua Hu, Huazhu Fu, Pengfei Zhu, Xiaochun Cao, Latent Multi-View Subspace Clustering , CVPR 2017.

Multi-View Complete Representation Generalized Latent Multi-View Subspace Learning Degradation networks mimicking data transmitting • General Correlation • Complete Representation • Deep “ Matrix Factorization ” Degradation networks [TPAMI’18] Changqing Zhang, Huazhu Fu, Qinghua Hu, Xiaochun Cao, Yuan Xie, Dacheng Tao, Dong Xu, Generalized Latent Multi-View Subspace Clustering , IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI) 2018.

Multi-View Complete Representation Generalized Latent Multi-View Subspace Learning Subspace representation Degradation networks [TPAMI’18] Changqing Zhang, Huazhu Fu, Qinghua Hu, Xiaochun Cao, Yuan Xie, Dacheng Tao, Dong Xu, Generalized Latent Multi-View Subspace Clustering , IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI) 2018.

Multi-View Complete Representation Generalized Latent Multi-View Subspace Learning Effectiveness of gLMSC in integrating Multiple Views Performance by using each single view and multiple views [TPAMI’18] Changqing Zhang, Huazhu Fu, Qinghua Hu, Xiaochun Cao, Yuan Xie, Dacheng Tao, Dong Xu, Generalized Latent Multi-View Subspace Clustering , IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI) 2018.

Multi-View Complete Representation CPM-Nets: Cross Partial Multi-View Networks Challenges of Classification on Partial Multi-View Data  多 视图 数据的缺失情况比 较 复 杂 ，如何避免 预处 理或者人工干 预 （如： 预 先 补 全 / 数据 丢 弃 / 根据缺失情况分 组 ）？  特征 类 型 / 模 态 种 类 多，不同 样 本缺失的模 态 不同（ 组 合 问题 ）；  甚至存在 test 样 本的缺失模式与所有 training 样 本的不同；  如何在理 论 上保 证 信息利用的充分性？（信息的完 备 性）  如何使得分 类 器具有更好的泛化性（特 别 是小 样 本情况）？

Multi-View Complete Representation CPM-Nets: Cross Partial Multi-View Networks Our Algorithm for Classification on Partial Multi-View Data 1. 自适 应 复 杂 缺失情况 2. 统 一表示的信息完 备 性：理 论 保 证 3. 统 一表示 结 构化： 简 化分 类 器 + 可解 释 性 1. Flexibility: Samples with arbitrary view-missing patterns; 2. Complete-Representation: Compact with full information; 3. Structured-Representation: Simplify classifier for interpretability; [ NeurIPS’19/Spotlight ] Changqing Zhang, Zongbo Han,Yajie Cui, Huazhu Fu, Joey Tianyi Zhou, Qinghua Hu, CPM-Nets: Cross Partial Multi-View Networks , Neural Information Processing Systems (NeurIPS) 2019.

Multi-View Complete Representation CPM-Nets: Cross Partial Multi-View Networks 反向 编码 ： Framework of CPM-Nets 保 证 信息完 备 + 自适 应 缺失 统 一表示 结 构 化： 简 化分 类 器 + 可解 释 性 [ NeurIPS’19/Spotlight ] Changqing Zhang, Zongbo Han,Yajie Cui, Huazhu Fu, Joey Tianyi Zhou, Qinghua Hu, CPM-Nets: Cross Partial Multi-View Networks , Neural Information Processing Systems (NeurIPS) 2019.

Multi-View Complete Representation CPM-Nets: Cross Partial Multi-View Networks Framework of CPM-Nets clustering-like 监 督 所有 观测 到的 视图 损 失函数： 统 一表 （ partial views ） 示 结 构化、无参化 编码进统 一表示 结 构化表示 信息完 备 性表示 [ NeurIPS’19/Spotlight ] Changqing Zhang, Zongbo Han,Yajie Cui, Huazhu Fu, Joey Tianyi Zhou, Qinghua Hu, CPM-Nets: Cross Partial Multi-View Networks , Neural Information Processing Systems (NeurIPS) 2019.

Multi-View Complete Representation CPM-Nets: Cross Partial Multi-View Networks Theoretical Analysis [ NeurIPS’19/Spotlight ] Changqing Zhang, Zongbo Han,Yajie Cui, Huazhu Fu, Joey Tianyi Zhou, Qinghua Hu, CPM-Nets: Cross Partial Multi-View Networks , Neural Information Processing Systems (NeurIPS) 2019.

Multi-View Complete Representation CPM-Nets: Cross Partial Multi-View Networks Comparison under Different Missing Rate  CCA-based methods: CCA/Kernelized CCA/Deep CCA;  Matrix Factorization- based method: Deep MF;  Metric Learning Methods: LMNN/ITML. [ NeurIPS’19/Spotlight ] Changqing Zhang, Zongbo Han,Yajie Cui, Huazhu Fu, Joey Tianyi Zhou, Qinghua Hu, CPM-Nets: Cross Partial Multi-View Networks , Neural Information Processing Systems (NeurIPS) 2019.