deep neural architectures for structured output problems
play

Deep neural architectures for structured output problems Clment - PowerPoint PPT Presentation

Dec 20, 2023 •760 likes •1.37k views

Deep neural architectures for structured output problems Clment Chatelain Soufiane Belharbi soufiane.belharbi@litislab.fr clement.chatelain@insa-rouen.fr LITIS - INSA de Rouen Joint work with: J.Lerouge, R.Herault, S.Adam, R.Modzelewski,

  1. Deep neural architectures for structured output problems Clément Chatelain Soufiane Belharbi soufiane.belharbi@litislab.fr clement.chatelain@insa-rouen.fr LITIS - INSA de Rouen Joint work with: J.Lerouge, R.Herault, S.Adam, R.Modzelewski, F.Jardin, B.Labbe May 19, 2015 images/logos LITIS - INSA de Rouen Deep neural architectures for structured output problems

  2. Plan 1 Structured Output Problems 2 Input/Output Deep Architecture (IODA) 3 Application of IODA to medical image labeling 4 Application of IODA to Facial Landmark Detection 5 Conclusion 6 Future Work on IODA images/logos LITIS - INSA de Rouen Deep neural architectures for structured output problems 1/46

  3. Structured Output Problems Traditional Machine Learning Problems f : X → y Inputs X ∈ R d : any type of input Outputs y ∈ R for the task: classification, regression, . . . Machine Learning for Structured Output Problems f : X → Y Inputs X ∈ R d : any type of input Outputs Y ∈ R d ′ , d ′ > 1 a structured object ( dependencies ) See C. Lampert slides [3]. images/logos LITIS - INSA de Rouen Deep neural architectures for structured output problems 2/46

  4. Structured Output Problems Traditional Machine Learning Problems f : X → y Inputs X ∈ R d : any type of input Outputs y ∈ R for the task: classification, regression, . . . Machine Learning for Structured Output Problems f : X → Y Inputs X ∈ R d : any type of input Outputs Y ∈ R d ′ , d ′ > 1 a structured object ( dependencies ) See C. Lampert slides [3]. images/logos LITIS - INSA de Rouen Deep neural architectures for structured output problems 2/46

  5. Structured Output Problems Data = representation ( values ) + structure ( dependencies ) Text: part-of-speech tagging, translation speech ⇄ text Protein folding Image Structured data images/logos LITIS - INSA de Rouen Deep neural architectures for structured output problems 3/46

  6. Structured Output Problems Approaches that Deal with Structured Output Data ◮ Kernel based methods: Kernel Density Estimation (KDE) ◮ Discriminative methods: Structure output SVM ◮ Graphical methods: HMM, CRF , MRF , . . . Drawbacks Perform one single data transformation Difficult to deal with high dimensional data Ideal approach ◮ Structured output problems ◮ High dimension data ◮ Multiple data transformation (complex mapping functions) Deep neural networks? images/logos LITIS - INSA de Rouen Deep neural architectures for structured output problems 4/46

  7. Structured Output Problems Approaches that Deal with Structured Output Data ◮ Kernel based methods: Kernel Density Estimation (KDE) ◮ Discriminative methods: Structure output SVM ◮ Graphical methods: HMM, CRF , MRF , . . . Drawbacks Perform one single data transformation Difficult to deal with high dimensional data Ideal approach ◮ Structured output problems ◮ High dimension data ◮ Multiple data transformation (complex mapping functions) Deep neural networks? images/logos LITIS - INSA de Rouen Deep neural architectures for structured output problems 4/46

  8. Structured Output Problems Traditional Deep neural Network Input layer Hidden layer 1 Hidden layer 2 Hidden layer 3 Hidden layer 4 Output layer x 1 x 2 x 3 y 1 car x 4 y 2 bus y 3 x 5 bike x 6 x 7 ◮ High dimension data OK ◮ Multiple data transformation (complex mapping functions) OK ◮ Structured output problems NO images/logos LITIS - INSA de Rouen Deep neural architectures for structured output problems 5/46

  9. Input/Output Deep Architecture (IODA) Plan 1 Structured Output Problems 2 Input/Output Deep Architecture (IODA) 3 Application of IODA to medical image labeling 4 Application of IODA to Facial Landmark Detection 5 Conclusion 6 Future Work on IODA images/logos LITIS - INSA de Rouen Deep neural architectures for structured output problems 6/46

  10. Input/Output Deep Architecture (IODA) IODA: ◮ Incorporate the output structure by learning ◮ Discover hidden dependencies in the outputs Input layer Hidden layer 1 Hidden layer 2 Hidden layer 3 Hidden layer 4 Output layer x 1 y 1 x 2 y 2 x 3 y 3 x 4 y 4 Structured object x 5 y 5 x 6 y 6 y 7 x 7 images/logos LITIS - INSA de Rouen Deep neural architectures for structured output problems 7/46

  11. Input/Output Deep Architecture (IODA) Training IODA Input layer Hidden layer 1 x 1 x 2 x 3 x 4 x 5 x 6 x 7 images/logos LITIS - INSA de Rouen Deep neural architectures for structured output problems 8/46

  12. Input/Output Deep Architecture (IODA) Training IODA Input layer Hidden layer 1 x 1 x 2 x 3 x 4 x 5 x 6 x 7 images/logos LITIS - INSA de Rouen Deep neural architectures for structured output problems 8/46

  13. Input/Output Deep Architecture (IODA) Training IODA Input layer Hidden layer 1 Hidden layer 2 x 1 x 2 x 3 x 4 x 5 x 6 x 7 images/logos LITIS - INSA de Rouen Deep neural architectures for structured output problems 9/46

  14. Input/Output Deep Architecture (IODA) Training IODA Input layer Hidden layer 1 Hidden layer 2 x 1 x 2 x 3 x 4 x 5 x 6 x 7 R in ( x i ; θ in ) = ˆ x i images/logos LITIS - INSA de Rouen Deep neural architectures for structured output problems 10/46

  15. Input/Output Deep Architecture (IODA) Training IODA Input layer Hidden layer 1 Hidden layer 2 Output layer x 1 y 1 y 2 x 2 x 3 y 3 x 4 y 4 x 5 y 5 x 6 y 6 x 7 y 7 R in ( x i ; θ in ) = ˆ x i images/logos LITIS - INSA de Rouen Deep neural architectures for structured output problems 11/46

  16. Input/Output Deep Architecture (IODA) Training IODA Input layer Hidden layer 1 Hidden layer 2 Hidden layer 4 Output layer x 1 y 1 x 2 y 2 x 3 y 3 x 4 y 4 x 5 y 5 x 6 y 6 y 7 x 7 R in ( x i ; θ in ) = ˆ x i images/logos LITIS - INSA de Rouen Deep neural architectures for structured output problems 12/46

  17. Input/Output Deep Architecture (IODA) Training IODA Input layer Hidden layer 1 Hidden layer 2 Hidden layer 3 Hidden layer 4 Output layer x 1 y 1 x 2 y 2 x 3 y 3 x 4 y 4 x 5 y 5 x 6 y 6 x 7 y 7 R in ( x i ; θ in ) = ˆ x i images/logos LITIS - INSA de Rouen Deep neural architectures for structured output problems 13/46

  18. Input/Output Deep Architecture (IODA) Training IODA Input layer Hidden layer 1 Hidden layer 2 Hidden layer 3 Hidden layer 4 Output layer x 1 y 1 x 2 y 2 x 3 y 3 x 4 y 4 x 5 y 5 x 6 y 6 x 7 y 7 R in ( x i ; θ in ) = ˆ R out ( y i ; θ out ) = ˆ x i y i images/logos LITIS - INSA de Rouen Deep neural architectures for structured output problems 14/46

  19. Input/Output Deep Architecture (IODA) Training IODA Input layer Hidden layer 1 Hidden layer 2 Hidden layer 3 Hidden layer 4 Output layer x 1 y 1 x 2 y 2 x 3 y 3 x 4 y 4 Back-propagation x 5 y 5 x 6 y 6 x 7 y 7 R in ( x i ; θ in ) = ˆ R out ( y i ; θ out ) = ˆ x i y i M ( x i ; θ , θ in , θ out ) = ˆ y i images/logos LITIS - INSA de Rouen Deep neural architectures for structured output problems 15/46

  20. Input/Output Deep Architecture (IODA) IODA framework: min θ L ( θ , D ( x , y )) � n L ( θ , D ( x , y )) = 1 � C ( M ( x i ; θ , θ in , θ out ) , y i ) n � �� � i = 1 Learn ( input → output ) dependencies) + ℓ in ( R in ( x i ; θ in ) , x i ) � �� � Learn input dependencies � + ℓ out ( R out ( y i ; θ out ) , y i ) � �� � Learn output dependencies C ( . ) , ℓ in ( . ) , ℓ out ( . ) : defined costs. min θ L ( θ , D ( x , y )) is hard to solve ⇒ split L ( θ , D ( x , y )) images/logos LITIS - INSA de Rouen Deep neural architectures for structured output problems 16/46

  21. Input/Output Deep Architecture (IODA) IODA framework: min θ L ( θ , D ( x , y )) � n L ( θ , D ( x , y )) = 1 � C ( M ( x i ; θ , θ in , θ out ) , y i ) n � �� � i = 1 Learn ( input → output ) dependencies) + ℓ in ( R in ( x i ; θ in ) , x i ) � �� � Learn input dependencies � + ℓ out ( R out ( y i ; θ out ) , y i ) � �� � Learn output dependencies C ( . ) , ℓ in ( . ) , ℓ out ( . ) : defined costs. min θ L ( θ , D ( x , y )) is hard to solve ⇒ split L ( θ , D ( x , y )) images/logos LITIS - INSA de Rouen Deep neural architectures for structured output problems 16/46

  22. Input/Output Deep Architecture (IODA) Relaxed-simplified version of IODA Unsupervised training: 1 � n 1 → Input dependencies : min i = 1 ℓ in ( R in ( x i ; θ in ) , x i ) n θ in 1 � n → output dependencies: min i = 1 ℓ out ( R out ( y i ; θ out ) , y i ) n θ out Standard supervised learning: 2 � n 1 min i = 1 C ( M ( x i ; θ , θ in , θ out ) , y i ) n θ , θ in , θ out Open source implementation Implemented using our library: Crino [1] [ Python-Theano based] . images/logos LITIS - INSA de Rouen Deep neural architectures for structured output problems 17/46

  23. Input/Output Deep Architecture (IODA) Relaxed-simplified version of IODA Unsupervised training: 1 � n 1 → Input dependencies : min i = 1 ℓ in ( R in ( x i ; θ in ) , x i ) n θ in 1 � n → output dependencies: min i = 1 ℓ out ( R out ( y i ; θ out ) , y i ) n θ out Standard supervised learning: 2 � n 1 min i = 1 C ( M ( x i ; θ , θ in , θ out ) , y i ) n θ , θ in , θ out Open source implementation Implemented using our library: Crino [1] [ Python-Theano based] . images/logos LITIS - INSA de Rouen Deep neural architectures for structured output problems 17/46

Recommend

Deep neural networks and structured output problems presentation of my current PhD work ISP

Deep neural networks and structured output problems presentation of my current PhD work ISP

Deep neural networks and structured output problems presentation of my current PhD work ISP seminar. UCL, Louvain-la-Neuve 2016 Soufiane Belharbi Romain Hrault Clment Chatelain Sbastien Adam soufiane.belharbi@insa-rouen.fr LITIS lab.,

1.27k views • 93 slides
Concise Introduction to Deep Neural Networks Outline: Classification problems Motivating

Concise Introduction to Deep Neural Networks Outline: Classification problems Motivating

Concise Introduction to Deep Neural Networks Outline: Classification problems Motivating Deep (large) Neural Network (DNN) classifiers Neurons and DNN architectures Numerical training of DNNs (supervised deep learning) Spiking

630 views • 33 slides
Neural Map Structured Memory for Deep RL Emilio Parisotto eparisot@andrew.cmu.edu PhD Student

Neural Map Structured Memory for Deep RL Emilio Parisotto eparisot@andrew.cmu.edu PhD Student

Neural Map Structured Memory for Deep RL Emilio Parisotto eparisot@andrew.cmu.edu PhD Student Machine Learning Department Carnegie Mellon University Supervised Learning Deep Most deep learning problems are posed as supervised Neural

562 views • 53 slides
Optimizing Deep Neural Networks Leena Chennuru Vankadara 26-10-2015 Table of Contents Neural

Optimizing Deep Neural Networks Leena Chennuru Vankadara 26-10-2015 Table of Contents Neural

Optimizing Deep Neural Networks Leena Chennuru Vankadara 26-10-2015 Table of Contents Neural Networks and loss surfaces Problems of Deep Architectures Optimization in Neural Networks Under fitting Proliferation of Saddle

1.67k views • 41 slides
Complex Prediction Problems A novel approach to multiple Structured Output Prediction Yasemin

Complex Prediction Problems A novel approach to multiple Structured Output Prediction Yasemin

Complex Prediction Problems A novel approach to multiple Structured Output Prediction Yasemin Altun Max-Planck Institute ECML HLIE08 Yasemin Altun Complex Prediction Information Extraction Extract structured information from unstructured

826 views • 27 slides
Neural Map: Structured Memory for Deep Reinforcement Learning Emilio Parisotto and Ruslan

Neural Map: Structured Memory for Deep Reinforcement Learning Emilio Parisotto and Ruslan

CS 885 Reinforcement Learning Neural Map: Structured Memory for Deep Reinforcement Learning Emilio Parisotto and Ruslan Salakhutdinov, ICLR 2018 Presented by Andreas Stckel July 6, 2018 motivation: navigating partially observable

1.49k views • 53 slides
Unsupervised Learning and Inverse Problems with Deep Neural Networks J oan Bruna, Stphane

Unsupervised Learning and Inverse Problems with Deep Neural Networks J oan Bruna, Stphane

Unsupervised Learning and Inverse Problems with Deep Neural Networks J oan Bruna, Stphane Mallat, Ivan Dokmanic, Martin de Hoop cole Normale Suprieure www.di.ens.fr/data Deep Convolutional Networks ( x ) x ( u ) L j L 1 f M (

520 views • 47 slides
Deep Residual Output Layers for Neural Language Generation Nikolaos Pappas, James Henderson June

Deep Residual Output Layers for Neural Language Generation Nikolaos Pappas, James Henderson June

Deep Residual Output Layers for Neural Language Generation Nikolaos Pappas, James Henderson June 13, 2019 Neural language generation Cat sat on ? Probability distribution at time t given context vector h t R d , weights W R d |V|

474 views • 14 slides
8. Other Deep Architectures CS 519 Deep Learning, Winter 2018 Fuxin Li With materials from Zsolt

8. Other Deep Architectures CS 519 Deep Learning, Winter 2018 Fuxin Li With materials from Zsolt

8. Other Deep Architectures CS 519 Deep Learning, Winter 2018 Fuxin Li With materials from Zsolt Kira and Ian Goodfellow A brief overview of other architectures Unsupervised Architectures Deep Belief Networks Autoencoders GANs

439 views • 25 slides
Neural Information Retrieval Wassila Lalouani 1 Plan Neural network architectures Neural

Neural Information Retrieval Wassila Lalouani 1 Plan Neural network architectures Neural

Neural Information Retrieval Wassila Lalouani 1 Plan Neural network architectures Neural IR tasks Neural IR architecture Feature Representations Neural IR query auto completion Neural IR query suggestion Neural IR document

1.48k views • 18 slides
Learning Deep Architectures Using Kernel Modules Li Deng Microsoft Research, Redmond (thanks

Learning Deep Architectures Using Kernel Modules Li Deng Microsoft Research, Redmond (thanks

MLSLP 2012 Learning Deep Architectures Using Kernel Modules Li Deng Microsoft Research, Redmond (thanks collaborations/discussions with many people) Introduction Deep neural net (modern multilayer perceptron) Hard to

330 views • 15 slides
Training Neural Networks: Normalization, Regularization etc. Intro to Deep Learning, Fall 2020 1

Training Neural Networks: Normalization, Regularization etc. Intro to Deep Learning, Fall 2020 1

Training Neural Networks: Normalization, Regularization etc. Intro to Deep Learning, Fall 2020 1 Quick Recap: Training a network Divergence between desired output and Average over all actual output of net for a given input training instances

1.75k views • 140 slides
Introduction to Deep Neural Networks 0. Logistics Spring 2020 1 Neural Networks are taking

Introduction to Deep Neural Networks 0. Logistics Spring 2020 1 Neural Networks are taking

Introduction to Deep Neural Networks 0. Logistics Spring 2020 1 Neural Networks are taking over! Neural networks have become one of the major thrust areas recently in various pattern recognition, prediction, and analysis problems In

987 views • 40 slides
Training Strategies CS 6355: Structured Prediction 1 So far we saw What is structured output

Training Strategies CS 6355: Structured Prediction 1 So far we saw What is structured output

Training Strategies CS 6355: Structured Prediction 1 So far we saw What is structured output prediction? Different ways for modeling structured prediction Conditional random fields, factor graphs, constraints What we only

986 views • 60 slides
What can Statistical Machine Translation teach Neural Machine Translation about Structured

What can Statistical Machine Translation teach Neural Machine Translation about Structured

What can Statistical Machine Translation teach Neural Machine Translation about Structured Prediction? Graham Neubig @ ICLR Workshop on Deep Reinforcement Learning Meets Structured Prediction 5/6/2019 Types of Prediction Types of

1.73k views • 127 slides
Tutorial on Neural Network Optimization Problems presentation by Ian Goodfellow Deep Learning

Tutorial on Neural Network Optimization Problems presentation by Ian Goodfellow Deep Learning

Tutorial on Neural Network Optimization Problems presentation by Ian Goodfellow Deep Learning Summer School Montreal August 9, 2015 Google Proprietary Optimization -Exhaustive search -Random search (genetic algorithms) -Analytical solution

1.1k views • 42 slides
Training Neural Networks: Optimization Intro to Deep Learning, Fall 2020 1 Quick Recap

Training Neural Networks: Optimization Intro to Deep Learning, Fall 2020 1 Quick Recap

Training Neural Networks: Optimization Intro to Deep Learning, Fall 2020 1 Quick Recap Gradient descent, Backprop 2 Quick Recap: Training a network Divergence between desired output and Average over all actual output of net for a given

1.21k views • 118 slides
Structured Output Learning with Indirect Supervision Ming-Wei Chang , Vivek Srikumar, Dan

Structured Output Learning with Indirect Supervision Ming-Wei Chang , Vivek Srikumar, Dan

Structured Output Learning with Indirect Supervision Ming-Wei Chang , Vivek Srikumar, Dan Goldwasser and Dan Roth Computer Science Department, University of Illinois at Urbana-Champaign Page. 1/31 Review: structured output prediction Example

1.41k views • 101 slides
Deep Learning with Neural Networks The Structure and Optimization of Deep Neural Networks Allan

Deep Learning with Neural Networks The Structure and Optimization of Deep Neural Networks Allan

Deep Learning with Neural Networks The Structure and Optimization of Deep Neural Networks Allan Zelener Machine Learning Reading Group January 7 th 2016 The Graduate Center, CUNY Objectives Explain some of the trends of deep learning and

680 views • 44 slides
Brain inspired Deep Learning Architectures Alex Movila Conventional artificial neural networks

Brain inspired Deep Learning Architectures Alex Movila Conventional artificial neural networks

Brain inspired Deep Learning Architectures Alex Movila Conventional artificial neural networks Inspired by the biological brain Benchmarked on tasks solved by the biological brain ...but compute in a fundamentally different

741 views • 71 slides
Course Information CS 6355: Structured Prediction Building up structured output prediction

Course Information CS 6355: Structured Prediction Building up structured output prediction

Course Information CS 6355: Structured Prediction Building up structured output prediction Refresher of binary classification Inference: Predicting structures, and introduction to multiclass complexity of inference and

213 views • 9 slides
Outline Convolutional Neural Network Architectures for Matching Natural Language Sentences.

Outline Convolutional Neural Network Architectures for Matching Natural Language Sentences.

Outline Hu, NIPS14 Irsoy, NIPS14 Outline Convolutional Neural Network Architectures for Matching Natural Language Sentences. NIPS14 Convolutional Sentence Model Convolutional Matching Models Experiments Deep Recursive Neural

779 views • 35 slides
Visualizing and Interpreting Deep Neural Networks Bolei Zhou Department of Information

Visualizing and Interpreting Deep Neural Networks Bolei Zhou Department of Information

Visualizing and Interpreting Deep Neural Networks Bolei Zhou Department of Information Engineering The Chinese University of Hong Kong Deep Neural Networks are Everywhere Playing Go Making Medical Decision Understanding Scenes Deep Neural

1.1k views • 46 slides
The Fundamentals of Deep Learning Building Blocks Theory with Applications Neural Units Neural

The Fundamentals of Deep Learning Building Blocks Theory with Applications Neural Units Neural

Deep Learning Antecedents Vision Case Study The Fundamentals of Deep Learning Building Blocks Theory with Applications Neural Units Neural Nets Deep Neural Nets Application ConvNets Jon Krohn LSTMs untapt jon@untapt.com

1.08k views • 106 slides

More recommend