towards understanding learning representations to what
play

Towards Understanding Learning Representations: To What Extent Do - PowerPoint PPT Presentation

Feb 19, 2024 •131 likes •473 views

. . . . . . . . . . . . . . Towards Understanding Learning Representations: To What Extent Do Difgerent Neural Networks Learn the Same Representation Liwei Wang Lunjia Hu Jiayuan Gu Yue Wu Zhiqiang Hu Kun He John Hopcroft .

  1. . . . . . . . . . . . . . . Towards Understanding Learning Representations: To What Extent Do Difgerent Neural Networks Learn the Same Representation Liwei Wang Lunjia Hu Jiayuan Gu Yue Wu Zhiqiang Hu Kun He John Hopcroft . . . . . . . . . . . . . . . . . . . . . . . . . . NeurIPS 2018 Spotlight

  2. In other words, are the learned representations commonly shared across multiple . . . . . . . . . . . . . . . . Motivation intermediate layers, and people design architectures in order to learn these representations better (e.g. CNN). However, there is a lack of theory on what these representations really are. One fundamental question: are the representations learned by deep nets robust? . . . . . . . . . . . . . . . . . . . . . . . . deep nets trained on the same task? ▶ It’s widely believed that deep nets learn particular features/representations in their

  3. In other words, are the learned representations commonly shared across multiple . . . . . . . . . . . . . . . . Motivation intermediate layers, and people design architectures in order to learn these representations better (e.g. CNN). One fundamental question: are the representations learned by deep nets robust? . . . . . . . . . . . . . . . . . . . . . . . . deep nets trained on the same task? ▶ It’s widely believed that deep nets learn particular features/representations in their ▶ However, there is a lack of theory on what these representations really are.

  4. . . . . . . . . . . . . . . . . . Motivation intermediate layers, and people design architectures in order to learn these representations better (e.g. CNN). In other words, are the learned representations commonly shared across multiple . . . . . . . . . . . . . . . . . . . . . . . deep nets trained on the same task? ▶ It’s widely believed that deep nets learn particular features/representations in their ▶ However, there is a lack of theory on what these representations really are. ▶ One fundamental question: are the representations learned by deep nets robust?

  5. . . . . . . . . . . . . . . Motivation Given a set of test examples, do the two deep nets share similarity in their output of layer i ? When layer i is the input layer, the similarity is high because both deep nets take the same test examples as input. When layer i is the fjnal output layer that predicts the classifjcation labels, the similarity is also high assuming both deep nets have tiny test error. How similar are intermediate layers? Do some groups of neurons in an intermediate layer learn features/representations . . . . . . . . . . . . . . . . . . . . . . . . . . that both deep nets share in common? How large are these groups? ▶ In particular, suppose we have two deep nets with the same architecture trained on the same training data but from difgerent initializations.

  6. . . . . . . . . . . . . . . . Motivation do the two deep nets share similarity in their output of layer i ? When layer i is the input layer, the similarity is high because both deep nets take the same test examples as input. When layer i is the fjnal output layer that predicts the classifjcation labels, the similarity is also high assuming both deep nets have tiny test error. How similar are intermediate layers? Do some groups of neurons in an intermediate layer learn features/representations . . . . . . . . . . . . . . . . . . . . . . . . . that both deep nets share in common? How large are these groups? ▶ In particular, suppose we have two deep nets with the same architecture trained on the same training data but from difgerent initializations. ▶ Given a set of test examples,

  7. . . . . . . . . . . . . . . . Motivation do the two deep nets share similarity in their output of layer i ? same test examples as input. When layer i is the fjnal output layer that predicts the classifjcation labels, the similarity is also high assuming both deep nets have tiny test error. How similar are intermediate layers? Do some groups of neurons in an intermediate layer learn features/representations . . . . . . . . . . . . . . . . . . . . . . . . . that both deep nets share in common? How large are these groups? ▶ In particular, suppose we have two deep nets with the same architecture trained on the same training data but from difgerent initializations. ▶ Given a set of test examples, ▶ When layer i is the input layer, the similarity is high because both deep nets take the

  8. . . . . . . . . . . . . . . . . Motivation do the two deep nets share similarity in their output of layer i ? same test examples as input. similarity is also high assuming both deep nets have tiny test error. How similar are intermediate layers? Do some groups of neurons in an intermediate layer learn features/representations . . . . . . . . . . . . . . . . . . . . . . . . that both deep nets share in common? How large are these groups? ▶ In particular, suppose we have two deep nets with the same architecture trained on the same training data but from difgerent initializations. ▶ Given a set of test examples, ▶ When layer i is the input layer, the similarity is high because both deep nets take the ▶ When layer i is the fjnal output layer that predicts the classifjcation labels, the

  9. . . . . . . . . . . . . . . . . Motivation do the two deep nets share similarity in their output of layer i ? same test examples as input. similarity is also high assuming both deep nets have tiny test error. Do some groups of neurons in an intermediate layer learn features/representations . . . . . . . . . . . . . . . . . . . . that both deep nets share in common? How large are these groups? . . . . ▶ In particular, suppose we have two deep nets with the same architecture trained on the same training data but from difgerent initializations. ▶ Given a set of test examples, ▶ When layer i is the input layer, the similarity is high because both deep nets take the ▶ When layer i is the fjnal output layer that predicts the classifjcation labels, the ▶ How similar are intermediate layers?

  10. . . . . . . . . . . . . . . . . . Motivation do the two deep nets share similarity in their output of layer i ? same test examples as input. similarity is also high assuming both deep nets have tiny test error. . . . . . . . . . . . . . . . . . . that both deep nets share in common? How large are these groups? . . . . . ▶ In particular, suppose we have two deep nets with the same architecture trained on the same training data but from difgerent initializations. ▶ Given a set of test examples, ▶ When layer i is the input layer, the similarity is high because both deep nets take the ▶ When layer i is the fjnal output layer that predicts the classifjcation labels, the ▶ How similar are intermediate layers? ▶ Do some groups of neurons in an intermediate layer learn features/representations

  11. . B there exist A and B such that a d , For test examples a 1 i after ReLU Output of layer Layer B B B B B X a i B B B B B Y X W Z i after ReLU Output of layer for all i , Y a i . Y a d Z W X Y We say activation vector of W W a d W a 1 activation vector of Z Z a d Z a 1 span activation vector of Y Y a 1 Z a i activation vector of X X a d X a 1 span B A Y a i X a i W a i Z a i W a i Layer A A . . . . . . . . . . . . . . . . . . . . . . . . . . . A . A A A A A A A A W Z Y X Two Groups of Neurons Learning the Same Representation: Exact Matches . . . . . . . . . . form an exact match! i + 1 i + 1

  12. . B For test examples a 1 i after ReLU Output of layer Layer B B B B B B B B there exist A and B such that B B Y X W Z i after ReLU Output of layer Layer . A A a d , for all i , A Y a 1 Z W X Y We say activation vector of W W a d W a 1 activation vector of Z Z a d Z a 1 span activation vector of Y Y a d activation vector of X X a i X a d X a 1 span B A Y a i X a i W a i Z a i W a i Z a i Y a i A A A . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A Y A A A A W Z form an exact match! X . . . . . Two Groups of Neurons Learning the Same Representation: Exact Matches i + 1 i + 1                

  13. . W B B B B B B B . Y X Z B i after ReLU Output of layer Layer A A A A A A A A B B A activation vector of Y Z W X Y We say activation vector of W W a d W a 1 activation vector of Z Z a d Z a 1 span Y a d Layer Y a 1 activation vector of X X a d X a 1 span B A for all i , there exist A and B such that i after ReLU Output of layer A B A . . . . . . . . . . . . . . . . . . . . . . . . . . . . . form an exact match! . Y . . . Two Groups of Neurons Learning the Same Representation: Exact Matches X . Z W . . . . For test examples a 1 , · · · , a d , i + 1 i + 1     X ( a i ) Z ( a i )         =             Y ( a i ) W ( a i )           Z ( a i ) X ( a i )         =       W ( a i ) Y ( a i )

Recommend

Learning Graph Representations for Video Understanding Xiaolong Wang Carnegie Mellon University

Learning Graph Representations for Video Understanding Xiaolong Wang Carnegie Mellon University

Learning Graph Representations for Video Understanding Xiaolong Wang Carnegie Mellon University Computer Vision Dog He et al. Mask R-CNN. ICCV 2017. Gler et al. DensePose: Dense Human Pose Estimation In The Wild. CVPR 2018. Deep Learning

941 views • 64 slides
Unraveling Meta-Learning: Understanding Feature Representations for Few-Shot Tasks Micah

Unraveling Meta-Learning: Understanding Feature Representations for Few-Shot Tasks Micah

Unraveling Meta-Learning: Understanding Feature Representations for Few-Shot Tasks Micah Goldblum, Steven Reich, Liam Fowl, Renkun Ni, Valeriia Cherepanova, Tom Goldstein University of Maryland, College Park, Maryland, USA

446 views • 6 slides
Understanding image representations by measuring their equivariance and equivalence Karel Lenc,

Understanding image representations by measuring their equivariance and equivalence Karel Lenc,

Visual Geometry Group, Department of Engineering Science Understanding image representations by measuring their equivariance and equivalence Karel Lenc, Andrea Vedaldi Representations for image understanding 2 image feature semantic

411 views • 30 slides
4. Number Representations You have a good understanding how a computer represents numbers. You

4. Number Representations You have a good understanding how a computer represents numbers. You

Educational Objectives 4. Number Representations You have a good understanding how a computer represents numbers. You can transform integers in binary representation and Domain of Types int , float and double Mixed Expressions perform

925 views • 10 slides
I-tutorial Learning of Invariant Representations in Sensory Cortex tomaso poggio CBMM McGovern

I-tutorial Learning of Invariant Representations in Sensory Cortex tomaso poggio CBMM McGovern

The Center for Brains, Minds and Machines I-tutorial Learning of Invariant Representations in Sensory Cortex tomaso poggio CBMM McGovern Institute, BCS, LCSL, CSAIL MIT I-theory Learning of Invariant Representations in Sensory Cortex

537 views • 50 slides
I-tutorial Learning of Invariant Representations in Sensory Cortex tomaso poggio CBMM McGovern

I-tutorial Learning of Invariant Representations in Sensory Cortex tomaso poggio CBMM McGovern

The Center for Brains, Minds and Machines I-tutorial Learning of Invariant Representations in Sensory Cortex tomaso poggio CBMM McGovern Institute, BCS, LCSL, CSAIL MIT I-theory Learning of Invariant Representations in Sensory Cortex

518 views • 40 slides
Lorentzian Distance Learning for Hyperbolic Representations Marc Law, Renjie Liao , Jake Snell,

Lorentzian Distance Learning for Hyperbolic Representations Marc Law, Renjie Liao , Jake Snell,

Lorentzian Distance Learning for Hyperbolic Representations Marc Law, Renjie Liao , Jake Snell, Richard Zemel June 13, 2019 University of Toronto, Vector Institute 1 Introduction: Hyperbolic Representations Manifolds with constant curvature:

968 views • 13 slides
Learning text representations from character-level data Grzegorz Chrupa la Department of

Learning text representations from character-level data Grzegorz Chrupa la Department of

Learning text representations from character-level data Grzegorz Chrupa la Department of Communication and Information Sciences Tilburg University CLIN 2013 Chrupa la (UvT) Text representations CLIN 2013 1 / 19 Text representations

221 views • 19 slides
Learning with Partially Ordered Representations Jonathan Rawski Department of Linguistics IACS

Learning with Partially Ordered Representations Jonathan Rawski Department of Linguistics IACS

Learning with Partially Ordered Representations Jonathan Rawski Department of Linguistics IACS Research Award Presentation August 11, 2018 The Main Idea Learning is eased when attributes of elements of sequences structure the space of

537 views • 29 slides
Unsupervised Le Learning of Video Representations using LS LSTMs Srivastava et al. University

Unsupervised Le Learning of Video Representations using LS LSTMs Srivastava et al. University

Unsupervised Le Learning of Video Representations using LS LSTMs Srivastava et al. University of Toronto Presented by Shyam Tailor The Overall Idea Take a sequence of images and encode into a fixed size latent representation Decode

210 views • 20 slides
12/28/2012 Learning Understanding the Standard All of life should be a learning experience,

12/28/2012 Learning Understanding the Standard All of life should be a learning experience,

12/28/2012 Learning Understanding the Standard All of life should be a learning experience, not of Care for Managing Sport- just for the trivial Related Concussions reasons but because by continuing the learning process, we are Thomas

361 views • 15 slides
Neural Networks Representations Fall 2017 Learning in the net Problem: Given a collection of

Neural Networks Representations Fall 2017 Learning in the net Problem: Given a collection of

Neural Networks Representations Fall 2017 Learning in the net Problem: Given a collection of input-output pairs, learn the function Learning for classification x 2 x 1 When the net must learn to classify.. Learning for classification x

960 views • 91 slides
CS7015 (Deep Learning) : Lecture 10 Learning Vectorial Representations Of Words Mitesh M. Khapra

CS7015 (Deep Learning) : Lecture 10 Learning Vectorial Representations Of Words Mitesh M. Khapra

CS7015 (Deep Learning) : Lecture 10 Learning Vectorial Representations Of Words Mitesh M. Khapra Department of Computer Science and Engineering Indian Institute of Technology Madras 1/1 Mitesh M. Khapra CS7015 (Deep Learning) : Lecture 10

1.06k views • 73 slides
SimCLR: A Simple Framework for Contrastive Learning of Visual Representations Ting Chen Simon

SimCLR: A Simple Framework for Contrastive Learning of Visual Representations Ting Chen Simon

SimCLR: A Simple Framework for Contrastive Learning of Visual Representations Ting Chen Simon Kornblith Mohammad Norouzi Geofgrey Hinton Google Research, Brain Team Unsupervised representation learning We tackle the problem of general visual

2.36k views • 33 slides
SOLAR: Deep Structured Representations for Model-Based Reinforcement Learning Marvin Zhang*,

SOLAR: Deep Structured Representations for Model-Based Reinforcement Learning Marvin Zhang*,

SOLAR: Deep Structured Representations for Model-Based Reinforcement Learning Marvin Zhang*, Sharad Vikram*, Laura Smith, Pieter Abbeel, Matthew J Johnson, Sergey Levine UC Berkeley, UC San Diego, Google Efficient reinforcement learning from

212 views • 20 slides
Neural Networks Representations Learning in the net Problem: Given a collection of

Neural Networks Representations Learning in the net Problem: Given a collection of

Neural Networks Representations Learning in the net Problem: Given a collection of input-output pairs, learn the function Learning for classification x 2 x 1 When the net must learn to classify.. Learn the classification boundaries

1.12k views • 94 slides
Learning Action Representations for Reinforcement Learning Georgios Scott Yash James Philip

Learning Action Representations for Reinforcement Learning Georgios Scott Yash James Philip

Learning Action Representations for Reinforcement Learning Georgios Scott Yash James Philip Theocharous Jordan Chandak Kostas Thomas Reinforcement Learning Problem Statement Thousands of possible actions! Problem Statement Thousands of

611 views • 31 slides
Learning Word Representations by Jointly Modeling Syntagmatic and Paradigmatic Relations Fei Sun ,

Learning Word Representations by Jointly Modeling Syntagmatic and Paradigmatic Relations Fei Sun ,

Learning Word Representations by Jointly Modeling Syntagmatic and Paradigmatic Relations Fei Sun , Jiafeng Guo, Yanyan Lan, Jun Xu, and Xueqi Cheng ofey.sunfei@gmail.com CAS Key Lab of Network Data Science and Technology Institute of Computing

1.02k views • 41 slides
Jointly Learning Word Representations and Composition Functions Using Predicate-Argument

Jointly Learning Word Representations and Composition Functions Using Predicate-Argument

Jointly Learning Word Representations and Composition Functions Using Predicate-Argument Structures Kazuma Hashimoto (UT) Pontus Stenetorp (UT) Makoto Miwa (TTI) Yoshimasa Tsuruoka (UT) U niversity of T okyo ( UT ) T oyota T echnological I

1.39k views • 119 slides
A Massively Scalable Architecture for Learning Representations from Heterogeneous Graphs NVIDIA

A Massively Scalable Architecture for Learning Representations from Heterogeneous Graphs NVIDIA

A Massively Scalable Architecture for Learning Representations from Heterogeneous Graphs NVIDIA GPU Technology Conference 2019 - San Jose, CA C. Bayan Bruss Anish Khazane 1. Overview & Background TODAYS TALK 2. Our Approach How to

680 views • 54 slides
Tractable Representations Inference Probabilistic Learning Models Applications Guy Van den

Tractable Representations Inference Probabilistic Learning Models Applications Guy Van den

Tractable Representations Inference Probabilistic Learning Models Applications Guy Van den Broeck University of California, Los Angeles based on joint AAAI-2020 and UAI-2019 tutorials with Antonio Vergari YooJung Choi University of

2.07k views • 177 slides
A Bayesian Approach to Finding Compact Representations for Reinforcement Learning Special thanks

A Bayesian Approach to Finding Compact Representations for Reinforcement Learning Special thanks

A Bayesian Approach to Finding Compact Representations for Reinforcement Learning Special thanks to Joelle Pineau for presenting our paper - July, 2012 1 Authors Alborz Geramifard Stefanie Tellex David Wingate Nicholas Roy Jonathan How 2

491 views • 25 slides
Transformation Equivariance vs. Invariance: Unsupervised Learning of Visual Representations

Transformation Equivariance vs. Invariance: Unsupervised Learning of Visual Representations

Transformation Equivariance vs. Invariance: Unsupervised Learning of Visual Representations Guo-Jun Qi guojunq@gmail.com Laboratory for MA chine P erception and LE arning ( MAPLE ) & Futurewei Technologies (Huawei Research USA) Laboratory

1.24k views • 58 slides
LEARNING REPRESENTATIONS OF SOURCE CODE FROM STRUCTURE & CONTEXT by Dylan Bourgeois

LEARNING REPRESENTATIONS OF SOURCE CODE FROM STRUCTURE & CONTEXT by Dylan Bourgeois

M.Sc. Thesis Defense 08.04.2019 LEARNING REPRESENTATIONS OF SOURCE CODE FROM STRUCTURE & CONTEXT by Dylan Bourgeois Supervised by Pr. Pierre Vandergheynst Michal Defferrard Pr. Jure Leskovec Dr. Michele Catasta 1 Introduction 2 Code:

786 views • 78 slides

More recommend