Manifold Mixup Alex Lamb, Vikas Verma, Christopher Beckham, Amir - PowerPoint PPT Presentation

Dec 09, 2022 •43 likes •163 views

Manifold Mixup Alex Lamb, Vikas Verma, Christopher Beckham, Amir Najafi, Ioannis Mitliagkas, David Lopez-Paz, Yoshua Bengio Troubling Properties of Deep Networks Issues with Current Methods Real data points occupy large volume in the

Manifold Mixup Alex Lamb*, Vikas Verma*, Christopher Beckham, Amir Najafi, Ioannis Mitliagkas, David Lopez-Paz, Yoshua Bengio
Troubling Properties of Deep Networks
Issues with Current Methods ● Real data points occupy large volume in the space ● Decision boundary is close to the data ● Data points from off the manifold occupy region overlapping with real data points
Improving Representations with Manifold Mixup Simple Algorithm - just a few lines of code ● ● Great Results Surprising Properties - backed by rigorous theory ●
Manifold Mixup - Simple Algorithm ● On each update, select a random layer uniformly (including the input). Sample 𝜇 ~ Beta( ⍺ , ⍺ ) ● Mix between two random examples from the minibatch at the selected ● layer with weights 𝜇 and (1- 𝜇 ). ● Mix the labels for those two examples in the same way to construct a soft target, yielding the manifold mixup loss , which compares the soft target with the output obtained with the mixed layer.
Manifold Mixup - Great Results Massive gains on likelihood Also works on SVHN, Tiny-Imagenet, Imagenet
Manifold Mixup - Great Results (external) ● Other labs have gotten great results with Manifold Mixup ● Handwriting Recognition (Moysset and Massina, ICDAR 2019) ● Convnets without Batch Normalization (Defazio & Bottou 2018) ● Prostate Cancer Segmentation with U-Net (Jung 2019)
Manifold Mixup - Surprising Properties Data Space Hidden Space
Manifold Mixup - Theory Justifying Properties ● When the manifold mixup loss is perfectly satisfied on a layer, the rest of the network becomes an implicit linear model, which we can call A. This can only be satisfied when dim(H) >= d - 1. ● ● The representations H have dim(H) - d + 1 degrees of freedom. Implications: fitting the manifold mixup objective exactly is feasible in later ● layers, and concentrates the representations such that they have zero measure.
What can Manifold Mixup do for you (applied)? ● Massively improved likelihood, so any classification task where you use the probabilities will probably be helped. Tasks with small amounts of labeled data ● ● May also help with outliers / out-of-distribution, but needs to be studied more
What can you do for Manifold Mixup (theory)? ● Our theory makes very precise assumptions, can these be relaxed? ● Is there a way to generalize mixing to multiple layers or to RNNs (and understand it)? ● Lots of broader work on spectral properties of learned representations: “An analytic theory of generalization dynamics and transfer learning in deep linear ○ networks” (Lampinen and Ganguli 2019) ● Would be great to explicitly connect to Manifold Mixup!
Questions? ● Also if you have any questions, are curious about applying Manifold Mixup, or want to collaborate, reach out to: vikasverma.iitm@gmail.com ○ ○ lambalex@iro.umontreal.ca

Recommend

Charting the Right Manifold: Manifold Mixup for Few-Shot Learning Puneet Mangla 1,2* , Mayank

Charting the Right Manifold: Manifold Mixup for Few-Shot Learning Puneet Mangla 1,2* , Mayank Singh 1* , Abhishek Sinha 1* , Nupur Kumari 1* , Balaji Krishnamurthy 1 , Vineeth N Balasubramanian 2 1 Media and Data Science Research, Adobe Inc. Noida,

1.22k views • 14 slides

Manifold Regularization Lorenzo Rosasco MIT, 9.520 L. Rosasco Manifold Regularization About

Manifold Regularization Lorenzo Rosasco MIT, 9.520 L. Rosasco Manifold Regularization About this class Goal To analyze the limits of learning from examples in high dimensional spaces. To introduce the semi-supervised setting and the use of

437 views • 31 slides

Linear Manifold Clustering Robert Haralick and Rave Harpaz Outline Background The linear

Linear Manifold Clustering Robert Haralick and Rave Harpaz Outline Background The linear manifold cluster model The Linear manifold clustering algorithm Linear manifold modeling Linear manifold subspace correlation clustering Conclusion

677 views • 67 slides

Mixup: A Development and Runtime Environment for Integration at the Presentation Layer Conference

See discussions, stats, and author profiles for this publication at: https://www.researchgate.net/publication/220940105 Mixup: A Development and Runtime Environment for Integration at the Presentation Layer Conference Paper July 2007 DOI:

282 views • 6 slides

Manifold Learning: Applications in Neuroimaging Robin Wolz 23/09/2011 Overview Manifold

Your own logo here Manifold Learning: Applications in Neuroimaging Robin Wolz 23/09/2011 Overview Manifold learning for Atlas Propagation Multi-atlas segmentation Challenges LEAP Manifold learning for classification

466 views • 29 slides

Manifold Regularization Lorenzo Rosasco 9.520 Class 10 March 6, 2011 L. Rosasco Manifold

Manifold Regularization Lorenzo Rosasco 9.520 Class 10 March 6, 2011 L. Rosasco Manifold Regularization About this class Goal To analyze the limits of learning from examples in high dimensional spaces. To introduce the semi-supervised

1.16k views • 31 slides

embedded in a 3-manifold is in- Def. A surface compressible if is injective.

Talk by Nathan Dunfield given at Univ of Warick, July 1999 embedded in a 3-manifold is in- Def. A surface compressible if is injective. Def. A 3-manifold is Haken if it is irreducible and contains an

260 views • 14 slides

Manifold Construction and Parameterization for Nonlinear Manifold-Based Model Reduction Chenjie

Manifold Construction and Parameterization for Nonlinear Manifold-Based Model Reduction Chenjie Gu and Jaijeet Roychowdhury {gcj,jr}@eecs.berkeley.edu University of California, Berkeley ASPDAC 2010 Slide 1 Outline Background Background

525 views • 31 slides

EMNLP | 2020 SeqMix: Augmenting Active Sequence Labeling via Sequence Mixup Rongzhi Zhang, Yue

EMNLP | 2020 SeqMix: Augmenting Active Sequence Labeling via Sequence Mixup Rongzhi Zhang, Yue Yu, Chao Zhang Georgia Institute of Technology Introduction Sequence labeling is core to many NLP tasks. Part-of-speech (POS) tagging.

698 views • 17 slides

Game Bot Identification Game Bot Identification based on Manifold Learning based on Manifold

Game Bot Identification Game Bot Identification based on Manifold Learning based on Manifold Learning Kuan Ta Chen Academia Sinica Hsing Kuo Pao NTUST Hong Chung Chang NTUST ACM NetGames 2008 Game Bots Game bots: automated AI programs

371 views • 36 slides

Manifold-driven spirals and rings Lia Athanassoula LAM, Marseille Lia Athanassoula Manifold

Manifold-driven spirals and rings Lia Athanassoula LAM, Marseille Lia Athanassoula Manifold driven spirals and rings Seoul October 2013 Figures here: P1: Romero-Gomez, Masdemont, Athanassoula,

201 views • 18 slides

A manifold structure on the set of functional observers Jochen Trumpf University of W urzburg

A manifold structure on the set of functional observers Jochen Trumpf University of W urzburg Math. Institute Germany A manifold structure on the set of functional observers p.1/12 Contents motivating problem A manifold structure on

756 views • 37 slides

Optimizing Physical Assets with Machine Learning Rajendra Koppula WWW.MANIFOLD.AI About Us

Optimizing Physical Assets with Machine Learning Rajendra Koppula WWW.MANIFOLD.AI About Us Manifold is a full-service AI development services firm that accelerates AI development for leading companies. Our team has a proven ability to design,

413 views • 38 slides

HIGH DIMENSIONAL MANIFOLD TOPOLOGY THEN AND NOW Andrew Ranicki (Edinburgh)

1 HIGH DIMENSIONAL MANIFOLD TOPOLOGY THEN AND NOW Andrew Ranicki (Edinburgh) http://www.maths.ed.ac.uk/ aar Orsay 7,8,9 December 2005 An n -dimensional topological manifold M is a paracompact Hausdorff topological space which is locally

1.17k views • 71 slides

n -dimensional manifold M with T := TM n -dimensional manifold M with T := TM T n -dimensional

n -dimensional manifold M with T := TM n -dimensional manifold M with T := TM T n -dimensional manifold M with T := TM T frame bundle GL ( R n , T ) is a principal GL ( n )-bundle n -dimensional manifold M with T := TM T + T frame bundle

1.34k views • 79 slides

Efficient Krylov Approximation for Manifold Learning Shinjae Yoo Computational Science

Efficient Krylov Approximation for Manifold Learning Shinjae Yoo Computational Science Initiative Outline Projects at BNL Big data and unsupervised learning Challenges of manifold learning in Big data Diverse Power Iteration

540 views • 23 slides

Image Masking Schemes for Local Manifold Learning Methods Marco F. Duarte Joint work with

Image Masking Schemes for Local Manifold Learning Methods Marco F. Duarte Joint work with Hamid Dadkhahi IEEE ICASSP - April 23 2015 Manifold Learning Given training points in , learn the mapping to the underlying K -dimensional

404 views • 22 slides

Manifold embedding for modeling spinal deformations Samuel Kadoury Philips Research North

Manifold embedding for modeling spinal deformations Samuel Kadoury Philips Research North America MICCAI 2011 Tutorial on Manifold Learning with Medical Images September 22 th , 2011 Spinal deformities Adolescent Idiopathic Scoliosis

746 views • 40 slides

Application of K ahler manifold to signal processing and Bayesian inference Jaehyung Choi ,

Review on K ahler manifold K ahlerian information geometry for signal processing Geometric shrinkage priors Example: ARFIMA Conclusion Application of K ahler manifold to signal processing and Bayesian inference Jaehyung Choi , 1 ,

264 views • 22 slides

Variational denoising for manifold-valued data Andreas Weinmann Helmholtz Center Munich & TU

TV minimization for manifold data Second order TV type functionals Potts and Blake-Zisserman functionals Variational denoising for manifold-valued data Andreas Weinmann Helmholtz Center Munich & TU M unchen Paris, le 21 novembre 2014

753 views • 44 slides

Linear Manifold Embeddings of Pattern Clusters Robert Haralick Rave Harpaz Pattern Recognition

Linear Manifolds The Algorithm Empirical Evaluation Linear Manifold Embeddings of Pattern Clusters Robert Haralick Rave Harpaz Pattern Recognition Laboratory The Graduate Center, City University of New York DIMACS 2005 Linear Manifold

596 views • 28 slides

D U E o i r ud ig el it i R o e t Riemannian Holonomy. To a Riemannian manifold ( M n

NONEMBEDDING AND NONEXTENSION RESULTS IN SPECIAL HOLONOMY ROBERT L. BRYANT DUKE UNIVERSITY D U E o i r ud ig el it i R o e t Riemannian Holonomy. To a Riemannian manifold ( M n , g ) associate its Levi-Civita connection , which

1.5k views • 112 slides

Medical Applica+ons for Manifold Learning: I)

Medical Applica+ons for Manifold Learning: I) Mul+-Modal Registra+on II) Breathing Ga+ng Chris+an Wachinger (wachinge@in.tum.de)

506 views • 39 slides

Fibration of the periodical eigenfunctions manifold into hypersurfaces Ya. Dymarskii Moscow,

Fibration of the periodical eigenfunctions manifold into hypersurfaces Ya. Dymarskii Moscow, MIPT, 2016 September 15 Ya. Dymarskii Fibration of the periodical eigenfunctions manifold into hypersurfaces The space of of self-adjoint periodic

728 views • 11 slides

Manifold Mixup Alex Lamb*, Vikas Verma*, Christopher Beckham, Amir - PowerPoint PPT Presentation

Manifold Mixup Alex Lamb*, Vikas Verma*, Christopher Beckham, Amir Najafi, Ioannis Mitliagkas, David Lopez-Paz, Yoshua Bengio Troubling Properties of Deep Networks Issues with Current Methods Real data points occupy large volume in the