neural encoding with structured decoding
play

Neural Encoding with Structured Decoding Pushpendre Rastogi 3 rd - PowerPoint PPT Presentation

Apr 12, 2023 •147 likes •625 views

Neural Encoding with Structured Decoding Pushpendre Rastogi 3 rd year CS Phd. Student pushpendre@jhu.edu Johns Hopkins University CLSP Student Seminar, Spring 2016 Pushpendre Rastogi (CLSP, JHU) Representations . . . 1 / 18 Outline 1

  1. Neural Encoding with Structured Decoding Pushpendre Rastogi 3 rd year CS Phd. Student pushpendre@jhu.edu Johns Hopkins University CLSP Student Seminar, Spring 2016 Pushpendre Rastogi (CLSP, JHU) Representations . . . 1 / 18

  2. Outline 1 Introduction 2 Best of Both Worlds: Neural Encoding with Structured Decoding 3 Acknowledgements and References Pushpendre Rastogi (CLSP, JHU) Representations . . . 2 / 18

  3. Introduction: Two Themes 1 Improving Neural Network Architectures. Pushpendre Rastogi (CLSP, JHU) Representations . . . 3 / 18

  4. Outline 1 Introduction 2 Best of Both Worlds: Neural Encoding with Structured Decoding 3 Acknowledgements and References Pushpendre Rastogi (CLSP, JHU) Representations . . . 4 / 18

  5. Background: What is the task? String transduction : Convert an input string to an output string. Example • Morphological Transduction: • Convert an imperative word in german to its past participle form. a b r e i b t �→ a b g e r i e b e n • Lemmatization: • Lemmatize a word in tagalog. b i n a w a l a n �→ b a w a l • Annotate a string: • Bob is a builder �→ Noun Verb Det Noun Pushpendre Rastogi (CLSP, JHU) Representations . . . 5 / 18

  6. What do we offer? Task = 13SIA Task = 2PIE 100 95 Accuracy 90 85 80 75 Task = 2PKE Task = rP 100 95 Accuracy 90 85 80 75 BiLSTM Seq2Seq BiLSTM Seq2Seq WFST Attention WFST Attention Method Method Pushpendre Rastogi (CLSP, JHU) Representations . . . 6 / 18

  7. The Idea Use a Neural Sequence Encoder to weight the arcs of a Weighted FST. Pushpendre Rastogi (CLSP, JHU) Representations . . . 7 / 18

  8. Background Weighted Finite State Transducers: Deterministic 3 0 1 2 s:s a:a y:y Pushpendre Rastogi (CLSP, JHU) Representations . . . 8 / 18

  9. Background Weighted Finite State Transducers: Deterministic 3 0 1 2 s:s a:a y:y Pushpendre Rastogi (CLSP, JHU) Representations . . . 8 / 18

  10. Background Weighted Finite State Transducers: Deterministic 3 0 1 2 s:s a:a y:y Pushpendre Rastogi (CLSP, JHU) Representations . . . 8 / 18

  11. Background Weighted Finite State Transducers: Deterministic 3 0 1 2 s:s a:a y:y What is a State? The States of an FST/WFST are its Memory. Previous Work weights this transducer. Pushpendre Rastogi (CLSP, JHU) Representations . . . 8 / 18

  12. Background Weighted Finite State Transducers: Non-Deterministic y:s ! :s ! :s s $ s:s y:y ! :a s:a a y a:s d: ! d:y i:y Pushpendre Rastogi (CLSP, JHU) Representations . . . 9 / 18

  13. Background Weighted Finite State Transducers: Non-Deterministic y:s ! :s ! :s s $ s:s y:y ! :a s:a a y a:s d: ! d:y i:y Pushpendre Rastogi (CLSP, JHU) Representations . . . 9 / 18

  14. Background Weighted Finite State Transducers: Non-Deterministic y:s ! :s ! :s s $ s:s y:y ! :a s:a a y a:s d: ! d:y i:y Pushpendre Rastogi (CLSP, JHU) Representations . . . 9 / 18

  15. Background Weighted Finite State Transducers: Non-Deterministic y:s ! :s ! :s s $ s:s y:y ! :a s:a a y a:s d: ! d:y i:y Pushpendre Rastogi (CLSP, JHU) Representations . . . 9 / 18

  16. Background Weighted Finite State Transducers: Non-Deterministic y:s What’s in a Path? A Path is an alignment. ! :s ( ǫ :s s:a a:s y:s) �→ say:sass ! :s s $ ( ǫ :s s:a a: ǫ y:y) �→ say:say s:s ( ǫ : ǫ s:s a:a y:y) �→ say:say y:y ! :a s:a ( ǫ :s s:a a:s y:y) �→ say:sasy Previous Work weights this a y a:s transducer. d: ! d:y i:y Pushpendre Rastogi (CLSP, JHU) Representations . . . 9 / 18

  17. Background Neural Bi-Directional Sequence Encoder α 0 e y e a e s Pushpendre Rastogi (CLSP, JHU) Representations . . . 10 / 18

  18. Background Neural Bi-Directional Sequence Encoder α 0 f ( α 0 , e s ) e y e a e s Pushpendre Rastogi (CLSP, JHU) Representations . . . 10 / 18

  19. Background Neural Bi-Directional Sequence Encoder α 0 α 1 e y e a e s Pushpendre Rastogi (CLSP, JHU) Representations . . . 10 / 18

  20. Background Neural Bi-Directional Sequence Encoder α 0 α 1 α 2 e y e a e s Pushpendre Rastogi (CLSP, JHU) Representations . . . 10 / 18

  21. Background Neural Bi-Directional Sequence Encoder α 0 α 1 α 2 e y e a e s β 3 Pushpendre Rastogi (CLSP, JHU) Representations . . . 10 / 18

  22. Background Neural Bi-Directional Sequence Encoder α 0 α 1 α 2 e y e a e s β 2 β 3 Pushpendre Rastogi (CLSP, JHU) Representations . . . 10 / 18

  23. Background Neural Bi-Directional Sequence Encoder α 0 α 1 α 2 e y e a e s β 1 β 2 β 3 Pushpendre Rastogi (CLSP, JHU) Representations . . . 10 / 18

  24. Background: Existing models. Weighted Finite State Transducers [Moh97, Eis02] Pros Cons Neural Encoders and Decoders [SVL14] Pros Cons Pushpendre Rastogi (CLSP, JHU) Representations . . . 11 / 18

  25. Background: Existing models. Weighted Finite State Transducers [Moh97, Eis02] Pros The states in an FST can be tailored for the task. Can compute the probability of a string. Cons Neural Encoders and Decoders [SVL14] Pros Cons Pushpendre Rastogi (CLSP, JHU) Representations . . . 11 / 18

  26. Background: Existing models. Weighted Finite State Transducers [Moh97, Eis02] Pros The states in an FST can be tailored for the task. Can compute the probability of a string. Cons Traditionally arcs weights are linear functionals of arc features. • ROI on feature engineering may be low. • The model may become slow if there are too many features. • The local features may not be expressive enough. Neural Encoders and Decoders [SVL14] Pros Cons Pushpendre Rastogi (CLSP, JHU) Representations . . . 11 / 18

  27. Background: Existing models. Weighted Finite State Transducers [Moh97, Eis02] Pros The states in an FST can be tailored for the task. Can compute the probability of a string. Cons Traditionally arcs weights are linear functionals of arc features. • ROI on feature engineering may be low. • The model may become slow if there are too many features. • The local features may not be expressive enough. Neural Encoders and Decoders [SVL14] Pros Produce reasonable results with zero feature engineering. Cons Pushpendre Rastogi (CLSP, JHU) Representations . . . 11 / 18

  28. Background: Existing models. Weighted Finite State Transducers [Moh97, Eis02] Pros The states in an FST can be tailored for the task. Can compute the probability of a string. Cons Traditionally arcs weights are linear functionals of arc features. • ROI on feature engineering may be low. • The model may become slow if there are too many features. • The local features may not be expressive enough. Neural Encoders and Decoders [SVL14] Pros Produce reasonable results with zero feature engineering. Cons Require a lot of training data for performance. Cannot return the probability of a string. Pushpendre Rastogi (CLSP, JHU) Representations . . . 11 / 18

  29. Neural Encoding with Structured Decoding 3 0 1 2 s:s a:a y:y Figure: The automaton I encoding say . y:s ! :s ! :s s $ s:s y:y ! :a s:a a y a:s d: ! d:y i:y Figure: Transducer F . Only a few of the possible states and edit arcs are shown. Previous Work weights these transducers Pushpendre Rastogi (CLSP, JHU) Representations . . . 12 / 18

  30. Neural Encoding with Structured Decoding ! :s ! :s ! :s ! :s 3 y:s 0 1 2 s:s a:s s:s a:a y:y s: ! a: ! y: ! 3, s Figure: The automaton I encoding say . 0, s 1, s 2, s s:a a:a y:a y:s ! :s ! :a ! :s ! :a ! :s ! :a ! :s ! :a ! :s ! :s s $ s:s s:s a:s y:s 0, a 1, a 2, a 3, a y:y s: ! a: ! y: ! ! :a s:a s:s a:s y:s a y a:s ! :s ! :s ! :s ! :s d: ! d:y i:y Figure: G = I ◦ F . Only a few states, Figure: Transducer F . Only a few of the but all arcs between them are shown. possible states and edit arcs are shown. Our Work weights this transducer. Previous Work weights these transducers Pushpendre Rastogi (CLSP, JHU) Representations . . . 12 / 18

  31. Neural Encoding with Structured Decoding ! :s ! :s ! :s ! :s 3 y:s 0 1 2 s:s a:s s:s a:a y:y s: ! a: ! y: ! 3, s Figure: The automaton I encoding say . 0, s 2, s 1, s s:a a:a y:a y:s ! :s ! :a ! :s ! :a ! :s ! :a ! :s ! :a ! :s ! :s s $ s:s s:s a:s y:s 0, a 1, a 2, a 3, a y:y s: ! a: ! y: ! ! :a s:a a s:s a:s y:s y a:s ! :s ! :s ! :s ! :s d: ! d:y i:y Figure: G = I ◦ F . Only a few states, Figure: Transducer F . Only a few of the but all arcs between them are shown. possible states and edit arcs are shown. Our Work weights this transducer. Previous Work weights these transducers Why do we do this? Weighting F ≡ Weighting edits per type. Weighting G ≡ Weighting edits per token. Neural features encode entire sentence. We get a context dependent output side language model. Pushpendre Rastogi (CLSP, JHU) Representations . . . 12 / 18

  32. Neural Encoding with Structured Decoding ! :s ! :s ! :s ! :s y:s s:s a:s α 0 α 1 α 2 s: ! a: ! y: ! e y 3, s e a 0, s 1, s 2, s e s s:a a:a y:a β 1 β 2 β 3 ! :s ! :a ! :s ! :a ! :s ! :a ! :s ! :a s:s a:s y:s 0, a 1, a 2, a 3, a s: ! a: ! y: ! s:s a:s y:s ! :s ! :s ! :s ! :s Figure: G = I ◦ F . Only a few states, but all arcs between them are shown. Our Work weights this transducer. Pushpendre Rastogi (CLSP, JHU) Representations . . . 12 / 18

Recommend

Overview Understanding the neural code Neural Encoding Encoding: Prediction of neural response to

Overview Understanding the neural code Neural Encoding Encoding: Prediction of neural response to

Overview Understanding the neural code Neural Encoding Encoding: Prediction of neural response to a given stimulus Decoding (homunculus): Given response, what was the stimulus? Mark van Rossum Given firing pattern, what will be the motor

191 views • 15 slides
Statistical models for neural encoding, decoding, information estimation, and optimal on-line

Statistical models for neural encoding, decoding, information estimation, and optimal on-line

Statistical models for neural encoding, decoding, information estimation, and optimal on-line stimulus design Liam Paninski Department of Statistics and Center for Theoretical Neuroscience Columbia University http://www.stat.columbia.edu/

725 views • 40 slides
Encoding and decoding neural information Encoding : building functional models of neurons/neural

Encoding and decoding neural information Encoding : building functional models of neurons/neural

CSE/NB 528 Final Lecture: All Good Things Must CSE/NB 528: Final Lecture 1 Course Summary Where have we been? Course Highlights Where do we go from here? Challenges and Open Problems Further Reading CSE/NB 528: Final

423 views • 18 slides
Why decoding? Understanding the neural code. Neural Decoding Given spikes, what was the

Why decoding? Understanding the neural code. Neural Decoding Given spikes, what was the

Why decoding? Understanding the neural code. Neural Decoding Given spikes, what was the stimulus? What aspects of the stimulus does the system encode? (capacity is limited) Mark van Rossum What information can be extracted from spike trains:

779 views • 16 slides
Statistical models for neural encoding, decoding, and optimal stimulus design Liam Paninski

Statistical models for neural encoding, decoding, and optimal stimulus design Liam Paninski

Statistical models for neural encoding, decoding, and optimal stimulus design Liam Paninski Department of Statistics and Center for Theoretical Neuroscience Columbia University http://www.stat.columbia.edu/ liam liam@stat.columbia.edu

178 views • 16 slides
Statistical models for neural encoding, decoding, and optimal stimulus design Liam Paninski

Statistical models for neural encoding, decoding, and optimal stimulus design Liam Paninski

Statistical models for neural encoding, decoding, and optimal stimulus design Liam Paninski Department of Statistics and Center for Theoretical Neuroscience Columbia University http://www.stat.columbia.edu/ liam liam@stat.columbia.edu

369 views • 21 slides
Bidirectional and executable specifications of machine code decoding and encoding Gang Tan, Penn

Bidirectional and executable specifications of machine code decoding and encoding Gang Tan, Penn

Bidirectional and executable specifications of machine code decoding and encoding Gang Tan, Penn State Univ. Joint work with Greg Morrisett, Cornell At LangSec Workshop , San Francesco, May 24 th , 2018 Machine Code Decoding/Encoding Binary

449 views • 26 slides
Bidirectional and executable specifications of machine code decoding and encoding Gang Tan, Penn

Bidirectional and executable specifications of machine code decoding and encoding Gang Tan, Penn

Bidirectional and executable specifications of machine code decoding and encoding Gang Tan, Penn State Univ. Joint work with Greg Morrisett, Cornell At LangSec Workshop , San Francesco, May 24 th , 2018 Machine Code Decoding/Encoding Binary

552 views • 33 slides
Beyond Sequential decoding toward parallel decoding In the context of neural sequence modelling

Beyond Sequential decoding toward parallel decoding In the context of neural sequence modelling

Beyond Sequential decoding toward parallel decoding In the context of neural sequence modelling Kyunghyun Cho New York University & Facebook AI Research Joint work with Jason Lee and Elman Mansimov Neural sequence modeling An arbitrary

461 views • 25 slides
STA and the encoding and decoding problems NEU 466M

STA and the encoding and decoding problems NEU 466M

STA and the encoding and decoding problems NEU 466M Instructor: Professor Ila R. Fiete Spring 2016 Last Gme: sGmulus, spike cross-correlaGon 5

844 views • 25 slides
Neural Encoding Models Maneesh Sahani maneesh@gatsby.ucl.ac.uk Gatsby Computational Neuroscience

Neural Encoding Models Maneesh Sahani maneesh@gatsby.ucl.ac.uk Gatsby Computational Neuroscience

Neural Encoding Models Maneesh Sahani maneesh@gatsby.ucl.ac.uk Gatsby Computational Neuroscience Unit University College London Term 1, Autumn 2011 Studying sensory systems x ( t ) y ( t ) x ( t )= G [ y ( t )] Decoding: (reconstruction)

812 views • 58 slides
Encoding/Decoding, Lower Bounds Russell Impagliazzo and Miles Jones Thanks to Janine Tiefenbruck

Encoding/Decoding, Lower Bounds Russell Impagliazzo and Miles Jones Thanks to Janine Tiefenbruck

Encoding/Decoding, Lower Bounds Russell Impagliazzo and Miles Jones Thanks to Janine Tiefenbruck http://cseweb.ucsd.edu/classes/sp16/cse21-bd/ May 9, 2016 Encoding: Fixed Density Strings Idea: use marker bit to indicate when to interpret output

1.1k views • 89 slides
Encoding/Decoding Russell Impagliazzo and Miles Jones Thanks to Janine Tiefenbruck

Encoding/Decoding Russell Impagliazzo and Miles Jones Thanks to Janine Tiefenbruck

Encoding/Decoding Russell Impagliazzo and Miles Jones Thanks to Janine Tiefenbruck http://cseweb.ucsd.edu/classes/sp16/cse21-bd/ May 9, 2016 Review: Terminology Rosen p. 407-413 A permutation of r elements from a set of n distinct objects is

1.07k views • 58 slides
Coarse-to-Fine Decoding for Neural Semantic Parsing July 16, 2018 Li Dong and Mirella Lapata

Coarse-to-Fine Decoding for Neural Semantic Parsing July 16, 2018 Li Dong and Mirella Lapata

Coarse-to-Fine Decoding for Neural Semantic Parsing July 16, 2018 Li Dong and Mirella Lapata Semantic Parsing Mapping natural language to structured representations Human-friendly -> Computer-friendly all flights from dallas before 10am

600 views • 28 slides
Encoding and Decoding Data Data is o(en encoded, or converted into a

Encoding and Decoding Data Data is o(en encoded, or converted into a

Encoding and Decoding Data Data is o(en encoded, or converted into a different form using a set of rules, and then decoded and changed back into its

368 views • 7 slides
Multilevel LDPC Lattices with Efficient Encoding and Decoding and a Generalization of Construction

Multilevel LDPC Lattices with Efficient Encoding and Decoding and a Generalization of Construction

Multilevel LDPC Lattices with Efficient Encoding and Decoding and a Generalization of Construction D Danilo Silva Paulo R. B. da Silva Department of Electrical and Electronic Engineering Federal University of Santa Catarina (UFSC), Brazil

755 views • 63 slides
Neural Decoding Matthias Hennig School of Informatics, University of Edinburgh January 2019 0

Neural Decoding Matthias Hennig School of Informatics, University of Edinburgh January 2019 0

Neural Decoding Matthias Hennig School of Informatics, University of Edinburgh January 2019 0 Acknowledgements: Mark van Rossum, Chris Williams and slides from Gatsby Liam Paninski. 1 / 54 Decoding brain activity Classi cation Which one?

1.07k views • 54 slides
Fast decoding in neural machine translation with Ray MAREK STRELEC Time cost statistics for

Fast decoding in neural machine translation with Ray MAREK STRELEC Time cost statistics for

Fast decoding in neural machine translation with Ray MAREK STRELEC Time cost statistics for decoding Zhang, Wen, et al. (2018) Ray q A flexible, high-performance distributed execution framework q Implements a dynamic task graph

105 views • 6 slides
McTiny: Encoding and decoding McEliece for tiny network servers 1978 McEliece public key:

McTiny: Encoding and decoding McEliece for tiny network servers 1978 McEliece public key:

1 2 McTiny: Encoding and decoding McEliece for tiny network servers 1978 McEliece public key: Daniel J. Bernstein, matrix G over F 2 . uic.edu , rub.de Normally m mG is injective. Tanja Lange, tue.nl Fundamental literature: 1962

2.16k views • 132 slides
Encoding/Decoding, Counting graphs Russell Impagliazzo and Miles Jones Thanks to Janine

Encoding/Decoding, Counting graphs Russell Impagliazzo and Miles Jones Thanks to Janine

Encoding/Decoding, Counting graphs Russell Impagliazzo and Miles Jones Thanks to Janine Tiefenbruck http://cseweb.ucsd.edu/classes/sp16/cse21-bd/ May 13, 2016 11-avoiding binary strings Lets consider the set of all n-bit binary strings with

1.08k views • 50 slides
I. Encoding and decoding planar curves Jeff Erickson University of Illinois, Urbana-Champaign

I. Encoding and decoding planar curves Jeff Erickson University of Illinois, Urbana-Champaign

One-Dimensional Computational Topology I. Encoding and decoding planar curves Jeff Erickson University of Illinois, Urbana-Champaign School on Low-Dimensional Geometry and Topology: Discrete and Algorithmic Aspects Institut Henri Poincar,

1.24k views • 106 slides
Neural Encoding Models Maneesh Sahani Gatsby Computational Neuroscience Unit University College

Neural Encoding Models Maneesh Sahani Gatsby Computational Neuroscience Unit University College

Neural Encoding Models Maneesh Sahani Gatsby Computational Neuroscience Unit University College London February 2019 Neural coding Neural Coding The brain appears to process sensory information in a modular way. Different structures and

1.74k views • 124 slides
Neural Decoding of Cursor Motion using Kalman Filter W. Wu, M. J. Black, Y. Gao, E. Bienenstock,

Neural Decoding of Cursor Motion using Kalman Filter W. Wu, M. J. Black, Y. Gao, E. Bienenstock,

Neural Decoding of Cursor Motion using Kalman Filter W. Wu, M. J. Black, Y. Gao, E. Bienenstock, M. Serruya, A. Shaikhouni, J. P. Donoghue NIPS 15, 2003 CSE 599E: Brain-Computer Interfaces Presented by: Jean Wu 19 April 2006 Overview [

306 views • 12 slides
Course Summary (thus far) F Neural Encoding What makes a neuron fire? (STA, covariance analysis)

Course Summary (thus far) F Neural Encoding What makes a neuron fire? (STA, covariance analysis)

CSE/NEUBEH 528 Modeling Synapses and Networks (Chapter 7) Image from Wikimedia Commons R. Rao, 528: Lecture 8 1 Lecture figures are from Dayan & Abbotts book Course Summary (thus far) F Neural Encoding What makes a neuron fire? (STA,

273 views • 12 slides

More recommend