Similarity of Neural Network Representations Revisited Simon - PowerPoint PPT Presentation

Mar 02, 2024 •414 likes •588 views

Similarity of Neural Network Representations Revisited Simon Kornblith, Mohammad Norouzi, Honglak Lee, Geofgrey Hinton Motivation We need tools to understand trained neural networks Neural network training involves interactions between an

Similarity of Neural Network Representations Revisited Simon Kornblith, Mohammad Norouzi, Honglak Lee, Geofgrey Hinton
Motivation We need tools to understand trained neural networks ● Neural network training involves interactions between an ○ algorithm and structured data We don’t know the structure of the data ○ One way to understand trained neural networks is by comparing their ● representations Similarity of Neural Network Representations Revisited
What is a Representation? (Centered) (Centered) Net A Features Net B Features Examples Examples Similarity of Neural Network Representations Revisited
Comparing Features = Comparing Examples Sum of squared dot products Dot product between reshaped inter-example (similarities) between features similarity matrices Similarity of Neural Network Representations Revisited
Comparing Features = Comparing Examples Similarity of Neural Network Representations Revisited
Comparing Features = Comparing Examples Centered kernel alignment (CKA) (Corues et al., 2012) RV-coeffjcient (Roberu & Escoufjer, 1976) Tucker’s congruence coeffjcient (Tucker, 1951) Similarity of Neural Network Representations Revisited
The Kernel Trick H is the centering matrix Similarity of Neural Network Representations Revisited P 7
A Sanity Check for Similarity Given two architecturally identical networks A and B trained from difgerent random initializations, a layer from net A should be most similar to the architecturally corresponding layer in net B conv1 conv2 conv3 conv4 conv5 conv6 conv7 conv8 avgpool conv1 conv2 conv3 conv4 conv5 conv6 conv7 conv8 avgpool Similarity of Neural Network Representations Revisited
A Sanity Check for Similarity Similarity of Neural Network Representations Revisited
A Sanity Check for Similarity Similarity of Neural Network Representations Revisited
CKA Reveals Network Pathology 1x Depth (94.1% on CIFAR-10) 2x Depth (95.1%) 4x Depth (93.2%) 8x Depth (91.9%) Similarity of Neural Network Representations Revisited P 11
CKA Reveals Network Pathology 1x Depth (94.1%) 2x Depth (95.0%) 4x Depth (93.2%) 8x Depth (91.9%) Similarity of Neural Network Representations Revisited P 12
CKA Reveals Network Pathology 1x Depth (94.1%) 2x Depth (95.0%) 4x Depth (93.2%) 8x Depth (91.9%) Similarity of Neural Network Representations Revisited P 13
CKA Reveals Network Pathology Similarity of Neural Network Representations Revisited
CKA Reveals Network Pathology Similarity of Neural Network Representations Revisited
Thank You! cka-similarity.github.io Similarity of Neural Network Representations Revisited

Recommend

Neural Information Retrieval Wassila Lalouani 1 Plan Neural network architectures Neural

Neural Information Retrieval Wassila Lalouani 1 Plan Neural network architectures Neural IR tasks Neural IR architecture Feature Representations Neural IR query auto completion Neural IR query suggestion Neural IR document

1.48k views • 18 slides

Neural network representations and visual processing in brains Katja Seeliger [ www.ccnlab.net |

Neural network representations and visual processing in brains Katja Seeliger [ www.ccnlab.net | Marcel van Gerven ] Neural network representations and visual processing in brains Computational Cognitive Neuroscience Lab Develop computational

405 views • 21 slides

Neural LMs Image: (Bengio et al, 03) One Hot Vectors Neural LMs (Bengio et al, 03)

Neural LMs Image: (Bengio et al, 03) One Hot Vectors Neural LMs (Bengio et al, 03) Low-dimensional Representations Learning representations by back-propagating errors Rumelhart, Hinton & Williams, 1986 A neural

1.64k views • 114 slides

Deep Neural Networks Reveal a Gradient in the Complexity of Neural Representations across the

Deep Neural Networks Reveal a Gradient in the Complexity of Neural Representations across the Ventral Stream Umut Gcl and Marcel A. J. van Gerven Article overview by Ilya Kuzovkin Computational Neuroscience Seminar University of Tartu

852 views • 63 slides

(Abstract) neural representations of spaces and concepts Neil Burgess Institute of Cognitive

UCL NEUROSCIENCE (Abstract) neural representations of spaces and concepts Neil Burgess Institute of Cognitive Neuroscience University College London SWC PhD program, October 2019 Abstract neural representations 1) Frames of reference for

897 views • 53 slides

Hard State Revisited: Network Filesystems Hard State Revisited: Network Filesystems Jeff Chase

Hard State Revisited: Network Filesystems Hard State Revisited: Network Filesystems Jeff Chase CPS 212, Fall 2000 Network File System (NFS) Network File System (NFS) server client syscall layer user programs VFS syscall layer NFS VFS

430 views • 29 slides

Delta-DNN : Efficiently Compressing Deep Neural Networks via Exploiting Floats Similarity The 49

Delta-DNN : Efficiently Compressing Deep Neural Networks via Exploiting Floats Similarity The 49 th International Conference on Parallel Processing (ICPP 2020) August 17-20, 2020 Outline Introduction Neural Networks Why compress

540 views • 21 slides

Sentiment Classification User Modeling with Neural Network for Review Rating Prediction. Duyu

User Behavior Analysis for Sentiment Classification User Modeling with Neural Network for Review Rating Prediction. Duyu Tang, Bing Qin, Ting Liu. IJCAI 2015, full paper. Learning Semantic Representations of Users and Products for Document

514 views • 16 slides

Beyond Credential Stuffing: Password Similarity Models using Neural Networks Bijeeta Pal*, Tal

Beyond Credential Stuffing: Password Similarity Models using Neural Networks Bijeeta Pal*, Tal Daniel + , Rahul Chatterjee*, and Thomas Ristenpart* *Cornell Tech + Technion 1 Password Breaches Millions of passwords leaked every year First half

369 views • 19 slides

Privacy-preserving Neural Representations of Text Maximin Coavoux Shashi Narayan Shay B.

Privacy-preserving Neural Representations of Text Maximin Coavoux Shashi Narayan Shay B. Cohen University of Edinburgh ILCC EMNLP 2018 Brussels 1 / 15 Context: Privacy and Neural Networks Machine learning uses data (e.g. UGC)

481 views • 19 slides

Artificial Neural Networks By: Kodi Neumiller Overview What is an artificial neural network

Artificial Neural Networks By: Kodi Neumiller Overview What is an artificial neural network Perceptron Feed forward Backpropagation Learning rate/momentum What is a neural network? Artificial Human Neurons Each

888 views • 12 slides

Machine Translation 2: Statistical MT: Neural MT and Representations Ondej Bojar

Machine Translation 2: Statistical MT: Neural MT and Representations Ondej Bojar bojar@ufal.mfg.cuni.cz Institute of Formal and Applied Linguistics Faculty of Mathematics and Physics Charles University, Prague May 2020 MT2: NMT and

703 views • 58 slides

Using Machine Learning to Study the Neural Representations of Language Meanings Tom M. Mitchell

Using Machine Learning to Study the Neural Representations of Language Meanings Tom M. Mitchell Carnegie Mellon University June 2017 How does neural activity encode word meanings? How does neural activity encode word meanings? How does

673 views • 64 slides

Same, Same But Different Recovering Neural Network Quantization Error Through Weight

Same, Same But Different Recovering Neural Network Quantization Error Through Weight Factorization Eldad Meller ICML 2019 Neural Network Quantization Quantization of Neural Networks is needed for efficient inference Quantization adds

482 views • 9 slides

Deep Learning Primer Nishith Khandwala Neural Networks Overview Neural Network Basics

Deep Learning Primer Nishith Khandwala Neural Networks Overview Neural Network Basics Activation Functions Stochastic Gradient Descent (SGD) Regularization (Dropout) Training Tips and Tricks Neural Network (NN)

828 views • 45 slides

Graph Neural Network Fang Yuanqiang, 2019/05/18 Graph Neural Network Why GNN? Preliminary

Graph Neural Network Fang Yuanqiang, 2019/05/18 Graph Neural Network Why GNN? Preliminary Fixed graph Vanilla Spectral Graph ConvNets ChebyNet GCN CayleyNet Multiple graphs Variable graph

817 views • 46 slides

Neural network applications ALVINN (Pomerleau, mid 1990s) Autonomous Land Vehicle in Neural

Neural network applications ALVINN (Pomerleau, mid 1990s) Autonomous Land Vehicle in Neural Network To date: Sharp Straight Sharp Left Ahead Right Neural networks: what are they 30 Output Units Backpropagation: efficient

346 views • 15 slides

Neural Networks for Machine Learning Lecture 2a An overview of the main types of neural network

Neural Networks for Machine Learning Lecture 2a An overview of the main types of neural network architecture Geoffrey Hinton with Nitish Srivastava Kevin Swersky Feed-forward neural networks These are the commonest type of neural

534 views • 32 slides

Investigating neural representations of spoken language Grzegorz Chrupaa In collaboration

Investigating neural representations of spoken language Grzegorz Chrupaa In collaboration with Afra Alishahi Lieke Gelderloos Marie Barking Mark van der Laan Automatic Speech Recognition A major success story in Language Technology

811 views • 41 slides

Semantic Similarity MultiJEDI ERC 259234 Semantic Similarity Semantic Similarity Mostly

SemEval 2014 Task-3 Cross-Level Semantic Similarity MultiJEDI ERC 259234 Semantic Similarity Semantic Similarity Mostly focused on similar types of lexical items Semantic Similarity What if we have different types of inputs? CLSS:

971 views • 47 slides

Towards Understanding Learning Representations: To What Extent Do Difgerent Neural Networks Learn

. . . . . . . . . . . . . . Towards Understanding Learning Representations: To What Extent Do Difgerent Neural Networks Learn the Same Representation Liwei Wang Lunjia Hu Jiayuan Gu Yue Wu Zhiqiang Hu Kun He John Hopcroft .

473 views • 33 slides

Towards Grounding Conceptual Spaces in Neural Representations Lucas Bechberger and Kai-Uwe

Towards Grounding Conceptual Spaces in Neural Representations Lucas Bechberger and Kai-Uwe Khnberger https://www.lucas-bechberger.de The Different Layers of Representation x :apple ( x ) red ( x ) Symbolic Layer Formal Logics

146 views • 13 slides

Neural representations of formulae A brief introduction Karel Chvalovsk CIIRC CTU

Neural representations of formulae A brief introduction Karel Chvalovsk CIIRC CTU Introduction the goal is to represent formulae by vectors (as good as possible) we have seen such a representation using hand-crafted features based on

682 views • 37 slides

Pre-trained Sentence and Contextualized Word Representations Graham Neubig Site

CS11-747 Neural Networks for NLP Pre-trained Sentence and Contextualized Word Representations Graham Neubig Site https://phontron.com/class/nn4nlp2020/ (w/ slides by Antonis Anastasopoulos) Remember: Neural Models I hate this movie

593 views • 43 slides