entropy and mutual information in models of deep neural
play

Entropy and mutual information in models of deep neural networks - PowerPoint PPT Presentation

Nov 15, 2022 •799 likes •1.16k views

Entropy and mutual information in models of deep neural networks NeurIPS 2018 - Thursday Dec 06th - Spotlight Poster @ Room 210 & 230 AB #110 Marylou Gabri (LPS ENS), Andre Manoel (INRIA Saclay, Owkin), Clment Luneau, Jean Barbier,

  1. Entropy and mutual information in models of deep neural networks NeurIPS 2018 - Thursday Dec 06th - Spotlight Poster @ Room 210 & 230 AB #110 Marylou Gabrié (LPS ENS), Andre Manoel (INRIA Saclay, Owkin), Clément Luneau, Jean Barbier, Nicolas Macris (EPFL), Lenka Zdeborová (CEA Saclay), Florent Krzakala (LPS ENS)

  2. Motivations Information theoretic arguments to deep learning Theory of generalization: Shwartz-Ziv et al. 2017, Saxe et al. 2018, Goldfeld et al. 2018 etc. • • New regularizer: Chalk et al. 2016, Alemi et al. 2017, Kolchinsky et al. 2017, Belghazi et al. 2017, Zhao et al. 2018, Achille et al. 2018, Hjelm et al. 2018, etc. Computing mutual informations in neural networks … is tricky • Intractable Computing or yet more involved • Sampled based estimations (non-parametric, variational methods ) Kraskov et al. 2014, Kolchinsky et al. 2017, Belghazi et al. 2017, Goldfeld et al. 2018 etc. Less and less reliable as networks get large è Need for benchmark controlled case è Here 1) Restrict to models of DNNS, 2) Leveverage statistical physics replica method

Recommend

Chapter 2 Entropy, Relative Entropy, and Mutual Infor- mation Peng-Hua Wang Graduate Institute

Chapter 2 Entropy, Relative Entropy, and Mutual Infor- mation Peng-Hua Wang Graduate Institute

Chapter 2 Entropy, Relative Entropy, and Mutual Infor- mation Peng-Hua Wang Graduate Institute of Communication Engineering National Taipei University Chapter Outline Chap. 2 Entropy, Relative Entropy, and Mutual Information 2.1 Entropy 2.2

752 views • 51 slides
Information Theory Lecture 1 Course introduction Entropy, relative entropy and mutual

Information Theory Lecture 1 Course introduction Entropy, relative entropy and mutual

Information Theory Lecture 1 Course introduction Entropy, relative entropy and mutual information: Cover & Thomas (CT) 2.15 Important inequalities: CT2.68, 2.10 Mikael Skoglund, Information Theory 1/26 Information Theory

160 views • 13 slides
Infotheory for Statistics and Learning Lecture 1 Entropy Relative entropy Mutual

Infotheory for Statistics and Learning Lecture 1 Entropy Relative entropy Mutual

Infotheory for Statistics and Learning Lecture 1 Entropy Relative entropy Mutual information f -divergence Mikael Skoglund 1/16 Entropy Over ( R , B ) , consider a discrete RV X with all probability in a countable set X B ,

538 views • 8 slides
Entropy and Shannon information Entropy and Shannon information For a random variable X with

Entropy and Shannon information Entropy and Shannon information For a random variable X with

Entropy and Shannon information Entropy and Shannon information For a random variable X with distribution p(x), the entropy is H[ X ] = - S x p( x ) log 2 p( x ) Information is defined as I[ X ] = - log 2 p( x ) Mutual information Typically,

695 views • 47 slides
Towards a Deep and Unified Understanding of Deep Neural Models in NLP Chaoyu Guan* 2 , Xiting

Towards a Deep and Unified Understanding of Deep Neural Models in NLP Chaoyu Guan* 2 , Xiting

Towards a Deep and Unified Understanding of Deep Neural Models in NLP Chaoyu Guan* 2 , Xiting Wang* 2 , Quanshi Zhang 1 , Runjin Chen 1 , Di He 2 , Xing Xie 2 * Equal Contribution 1 John Hopcroft Center and the MoE Key Lab of Artificial

219 views • 11 slides
Visualizing and Interpreting Deep Neural Networks Bolei Zhou Department of Information

Visualizing and Interpreting Deep Neural Networks Bolei Zhou Department of Information

Visualizing and Interpreting Deep Neural Networks Bolei Zhou Department of Information Engineering The Chinese University of Hong Kong Deep Neural Networks are Everywhere Playing Go Making Medical Decision Understanding Scenes Deep Neural

1.1k views • 46 slides
Mutual information deep regularization for semi- supervised segmentation J. Peng, M. Pedersoli,

Mutual information deep regularization for semi- supervised segmentation J. Peng, M. Pedersoli,

MIDL 2020 presentation Mutual information deep regularization for semi- supervised segmentation J. Peng, M. Pedersoli, C. Desrosiers Outline We proposed a semi-supervised segmentation method for medical image regularized by Mutual Information

256 views • 13 slides
Eff Efficient Entr icient Entropy opy Est Estimation imation for for Mutua Mutual l

Eff Efficient Entr icient Entropy opy Est Estimation imation for for Mutua Mutual l

Efficient entropy estimation for MIA using B-splines Eff Efficient Entr icient Entropy opy Est Estimation imation for for Mutua Mutual l Information Information Analys Analysis is using using B-splines splines Alexandre VENELLI IML

871 views • 25 slides
ECEN 5022 Cryptography Introduction to Information Theory Peter Mathys University of Colorado

ECEN 5022 Cryptography Introduction to Information Theory Peter Mathys University of Colorado

Entropy Mutual Information Perfect Cryptosystem ECEN 5022 Cryptography Introduction to Information Theory Peter Mathys University of Colorado Spring 2008 Peter Mathys ECEN 5022 Cryptography Entropy Entropy Mutual Information Binary

484 views • 12 slides
(part 1) CMSC 473/673 UMBC Some slides adapted from 3SLP Outline Maximum Entropy models

(part 1) CMSC 473/673 UMBC Some slides adapted from 3SLP Outline Maximum Entropy models

Maxent and Neural Language Models (part 1) CMSC 473/673 UMBC Some slides adapted from 3SLP Outline Maximum Entropy models Defining the model Defining the objective Learning: Optimizing the objective Math: gradient derivation Neural

1.9k views • 133 slides
NEURAL CONVERSATIONAL MODELS WITH ENTROPY-BASED DATA FILTERING Richard Csaky 1 , Patrik Purgai 1

NEURAL CONVERSATIONAL MODELS WITH ENTROPY-BASED DATA FILTERING Richard Csaky 1 , Patrik Purgai 1

IMPROVING NEURAL CONVERSATIONAL MODELS WITH ENTROPY-BASED DATA FILTERING Richard Csaky 1 , Patrik Purgai 1 , Gabor Recski 1,2 1 Budapest University of Technology 2 Sclable AI Introduction Takeaways Better responses by filtering training

410 views • 13 slides
Deep Neural Networks and Hidden Markov Models in i-vector-based Text-Dependent Speaker

Deep Neural Networks and Hidden Markov Models in i-vector-based Text-Dependent Speaker

Introduction HMM based method Deep Neural Networks (DNNs) Experiments and Results Conclusions Deep Neural Networks and Hidden Markov Models in i-vector-based Text-Dependent Speaker Verification Hossein Zeinali 1 , 2 , Luk s Burget 2 ,

301 views • 11 slides
Maxent Models (III), & Neural Language Models CMSC 473/673 UMBC September 25 th , 2017 Some

Maxent Models (III), & Neural Language Models CMSC 473/673 UMBC September 25 th , 2017 Some

Maxent Models (III), & Neural Language Models CMSC 473/673 UMBC September 25 th , 2017 Some slides adapted from 3SLP Recap from last time Maximum Entropy Models a more general language model argmax ) ()

1.46k views • 66 slides
Overview Coding and Information Theory What is information theory? Entropy Coding Chris

Overview Coding and Information Theory What is information theory? Entropy Coding Chris

Overview Coding and Information Theory What is information theory? Entropy Coding Chris Williams Rate-distortion theory School of Informatics, University of Edinburgh Mutual information Channel capacity November 2007 Reading: Bishop 1.6

389 views • 5 slides
Deep Learning with Neural Networks The Structure and Optimization of Deep Neural Networks Allan

Deep Learning with Neural Networks The Structure and Optimization of Deep Neural Networks Allan

Deep Learning with Neural Networks The Structure and Optimization of Deep Neural Networks Allan Zelener Machine Learning Reading Group January 7 th 2016 The Graduate Center, CUNY Objectives Explain some of the trends of deep learning and

680 views • 44 slides
Feature extraction from deep models Olgert Denas Synopsis Intro to deep models Applications

Feature extraction from deep models Olgert Denas Synopsis Intro to deep models Applications

Feature extraction from deep models Olgert Denas Synopsis Intro to deep models Applications Neurons & Nets dimer Learning & Depth G1E model Feature extraction Theory 1 Layer Nets Neural computation

1.32k views • 45 slides
Entropy, Relative Entropy, Cross Entropy Entropy Entropy, H(x) is a measure of the uncertainty of

Entropy, Relative Entropy, Cross Entropy Entropy Entropy, H(x) is a measure of the uncertainty of

Entropy, Relative Entropy, Cross Entropy Entropy Entropy, H(x) is a measure of the uncertainty of a discrete random variable. Properties: H(x) >= 0 Entropy Entropy Lesser the probability for an event, larger the entropy.

471 views • 21 slides
9. Sequential Neural Models CS 519 Deep Learning, Winter 2018 Fuxin Li With materials from

9. Sequential Neural Models CS 519 Deep Learning, Winter 2018 Fuxin Li With materials from

9. Sequential Neural Models CS 519 Deep Learning, Winter 2018 Fuxin Li With materials from Andrej Karpathy, Bo Xie, Zsolt Kira Sequential and Temporal Data Many applications exhibited by dynamically changing states Language (e.g.

1.12k views • 91 slides
Formal Modeling in Cognitive Science Lecture 25: Entropy, Joint Entropy, Conditional Entropy 1

Formal Modeling in Cognitive Science Lecture 25: Entropy, Joint Entropy, Conditional Entropy 1

Entropy Entropy Formal Modeling in Cognitive Science Lecture 25: Entropy, Joint Entropy, Conditional Entropy 1 Entropy Entropy and Information Joint Entropy Frank Keller Conditional Entropy School of Informatics University of Edinburgh

81 views • 4 slides
Deep Neural Networks for Data-Driven Turbulence Models @ICERM 2019: Scientific Machine Learning

Deep Neural Networks for Data-Driven Turbulence Models @ICERM 2019: Scientific Machine Learning

Institute of Aerodynamics and Gas Dynamics Andrea Beck David Flad C.-D. Munz Deep Neural Networks for Data-Driven Turbulence Models @ICERM 2019: Scientific Machine Learning Outline 1 Introduction Machine Learning with Neural Networks 2

627 views • 49 slides
Lecture 5: Measures of Information for Continuous Random Variables I-Hsiang Wang Department of

Lecture 5: Measures of Information for Continuous Random Variables I-Hsiang Wang Department of

Entropy and Mutual Information Differential Entropy Lecture 5: Measures of Information for Continuous Random Variables I-Hsiang Wang Department of Electrical Engineering National Taiwan University ihwang@ntu.edu.tw October 26, 2015 1 / 24

540 views • 24 slides
Deep Neural Ranking Models for Argument Retrieval Masters Thesis by Saeed Entezari Referees:

Deep Neural Ranking Models for Argument Retrieval Masters Thesis by Saeed Entezari Referees:

Deep Neural Ranking Models for Argument Retrieval Masters Thesis by Saeed Entezari Referees: Prof. Stein, PD. Dr. Jakoby Supervisor: Michael V olske Faculty of Media Bauhaus Universit at Weimar September 16, 2020 1/58 Agenda

777 views • 59 slides
Lecture 2 Measures of Information I-Hsiang Wang Department of Electrical Engineering National

Lecture 2 Measures of Information I-Hsiang Wang Department of Electrical Engineering National

Entropy and Conditional Entropy Mutual Information and KullbackLeibler Divergence Lecture 2 Measures of Information I-Hsiang Wang Department of Electrical Engineering National Taiwan University ihwang@ntu.edu.tw September 22, 2015 1 / 51

799 views • 51 slides
Lecture 2 Measures of Information I-Hsiang Wang Department of Electrical Engineering National

Lecture 2 Measures of Information I-Hsiang Wang Department of Electrical Engineering National

Entropy and Conditional Entropy Mutual Information and KullbackLeibler Divergence Lecture 2 Measures of Information I-Hsiang Wang Department of Electrical Engineering National Taiwan University ihwang@ntu.edu.tw September 16, 2015 1 / 42

554 views • 42 slides

More recommend