On the use of phone-gram units in recurrent neural networks for - PowerPoint PPT Presentation

Jan 05, 2023 •109 likes •166 views

On the use of phone-gram units in recurrent neural networks for language identification Christian Salamea, Luis F. D'Haro, Ricardo de Crdoba, Rubn San-Segundo Speech Technology Group. Dept. of Electronic Engineering. Universidad

On the use of phone-gram units in recurrent neural networks for language identification Christian Salamea, Luis F. D'Haro, Ricardo de Córdoba, Rubén San-Segundo Speech Technology Group. Dept. of Electronic Engineering. Universidad Politécnica de Madrid
Summary of the paper Phone-gram units in RNN Phonemes as input Using a 1-N codification (N the total number of phonemes) We’ve incorporated contextual information in the NN input: uniphones, diphones, and trigrams We propose the concatenation of n-adjacent phonemes RNNLM-P (phone grams) applied to LID: Based on a PPRLM architecture: For each phonetic recognizer, a phoneme sequence is obtained In evaluation, for each utterance, an entropy metric is obtained from the RNNLM The entropy scores are calibrated and fused Odyssey 2016 - Bilbao
Summary of the paper Phonetic Vector Representation For Vocabulary Reduction Use K-means to group phone-grams Neural Embedding Models with the Skip-Gram model We work at the phone level 7.3% improvement thanks to vocabulary reduction The most relevant RNN parameters are considered The Number of neurons in the state layer (NNE) Number of classes (NCS). Phone-grams are grouped in the output layer in a factorization process A high NCS value speeds up the RNN training but the final language model is less accurate Number of state layers (MEM) corresponding to previous times. Previous context information is taken into account Odyssey 2016 - Bilbao
Results System System Abs Abs MFCCs MFCCs 7,60 7,60 PPRLM PPRLM 11,57 11,57 RNNLM-P RNNLM-P 10,87 10,87 Cavg Cavg System System Abs Abs Improve % Improve % RNNLM-P+PPRLM RNNLM-P+PPRLM 10,51 10,51 9,2 9,2 PPRLM+MFCCs PPRLM+MFCCs 5,10 5,10 32,9 32,9 RNNLM-P+MFCCs RNNLM-P+MFCCs 5,04 5,04 33,7 33,7 RNNLM-P+PPRLM+MFCC RNNLM-P+PPRLM+MFCC 4,80 4,80 36,8 36,8 Odyssey 2016 - Bilbao

Recommend

Legendre Memory Units (LMUs) Continuous-Time Representation in Recurrent Neural Networks Aaron

Legendre Memory Units (LMUs) Continuous-Time Representation in Recurrent Neural Networks Aaron Voelker, Ivana Kaji, Chris Eliasmith Image caption Music composition generation LSTMs Speech recognition Disease prediction Machine

482 views • 15 slides

Recurrent Language Models CMSC 470 Marine Carpuat Toward a Neural Language Model Figures by

Recurrent Language Models CMSC 470 Marine Carpuat Toward a Neural Language Model Figures by Philipp Koehn (JHU) Count-based n-gram models vs. feedforward neural networks Pros of feedforward neural LM Word embeddings capture

547 views • 16 slides

Recurrent Language Models CMSC 470 Marine Carpuat Toward a Neural Language Model Figures by

279 views • 15 slides

CHAPTER II I CHAPTER I Recurrent Neural Networks Recurrent Neural Networks CHAPTER II : I :

Ugur HALICI - METU EEE - ANKARA 11/18/2004 CHAPTER II I CHAPTER I Recurrent Neural Networks Recurrent Neural Networks CHAPTER II : I : Recurrent Neural Networks CHAPTER I Recurrent Neural Networks Introduction In this chapter first the

404 views • 27 slides

Computa(on through dynamics Using recurrent neural networks to unveil mechanism in neural

Computa(on through dynamics Using recurrent neural networks to unveil mechanism in neural circuits David Sussillo with Valerio Mante and Bill Newsome Table of contents Introduc(on Training recurrent neural networks(RNNs) Understanding how

1.57k views • 105 slides

Recurrent Neural Networks CS60010: Deep Learning Abir Das IIT Kharagpur Mar 11, 2020

Recurrent Neural Networks CS60010: Deep Learning Abir Das IIT Kharagpur Mar 11, 2020 Introduction Recurrent Neural Network LSTM Agenda Get introduced to different recurrent neural architecture e.g. , RNNs, LSTMs, GRUs etc. Get

743 views • 48 slides

Understanding LSTM Networks Recurrent Neural Networks An unrolled recurrent neural network The

Understanding LSTM Networks Recurrent Neural Networks An unrolled recurrent neural network The Problem of Long-Term Dependencies RNN short-term dependencies Language model trying to predict the next word based on the previous ones the clouds

729 views • 27 slides

IN5550 Neural Methods in Natural Language Processing Applications of Recurrent Neural Networks

IN5550 Neural Methods in Natural Language Processing Applications of Recurrent Neural Networks Stephan Oepen University of Oslo March 24, 2019 Our Roadmap Last Week Language structure: sequences, trees, graphs Recurrent Neural

217 views • 19 slides

CSEP 517: Natural Language Processing Recurrent Neural Networks Autumn 2018 Luke Zettlemoyer

CSEP 517: Natural Language Processing Recurrent Neural Networks Autumn 2018 Luke Zettlemoyer University of Washington [most slides from Yejin Choi] RECURRENT NEURAL L NE NETWOR WORKS Recurrent Neural Networks (RNNs) Each input

787 views • 29 slides

CS6501: Deep Learning for Visual Recognition Recurrent Neural Networks (RNNs) Todays Class

CS6501: Deep Learning for Visual Recognition Recurrent Neural Networks (RNNs) Todays Class Recurrent Neural Network Cell Recurrent Neural Networks (RNNs) Bi-Directional Recurrent Neural Networks (Bi-RNNs) Multiple-layer /

583 views • 47 slides

CS6501: Deep Learning for Visual Recognition Recurrent Neural Networks (RNNs) Todays Class

793 views • 21 slides

IN5550 Neural Methods in Natural Language Processing Recurrent Neural Networks Stephan Oepen

IN5550 Neural Methods in Natural Language Processing Recurrent Neural Networks Stephan Oepen University of Oslo March 10, 2019 Our Roadmap Today Language structure: sequences, trees, graphs Recurrent Neural Networks Different

780 views • 76 slides

Recurrent Neural Networks Greg Mori - CMPT 419/726 Goodfellow, Bengio, and Courville: Deep

Recurrent Neural Networks Long Short-Term Memory Temporal Convolutional Networks Examples Recurrent Neural Networks Greg Mori - CMPT 419/726 Goodfellow, Bengio, and Courville: Deep Learning textbook Ch. 10 Recurrent Neural Networks Long

667 views • 18 slides

Lecture 9: Recurrent Neural Networks Princeton University COS 495 Instructor: Yingyu Liang

Deep Learning Basics Lecture 9: Recurrent Neural Networks Princeton University COS 495 Instructor: Yingyu Liang Introduction Recurrent neural networks Dates back to (Rumelhart et al. , 1986) A family of neural networks for handling

348 views • 34 slides

Introduction to Recurrent Neural Networks Jakob Verbeek Modeling sequential data with Recurrent

Introduction to Recurrent Neural Networks Jakob Verbeek Modeling sequential data with Recurrent Neural Networks Compact schematic drawing of standard multi-layer perceptron (MLP) output hidden input Modeling sequential data So far we

796 views • 50 slides

Recurrent Neural Networks (RNN) Artificial Intelligence @ Allegheny College Janyl Jumadinova

Recurrent Neural Networks (RNN) Artificial Intelligence @ Allegheny College Janyl Jumadinova March 9, 2020 Alex Graves, Supervised Sequence Labelling with Recurrent Neural Networks http://colah.github.io/posts/2015-08-Understanding-LSTMs/

538 views • 18 slides

Recurrent Neural Network Rachel Hu and Zhi Zhang Amazon AI d2l.ai Outline Dependent Random

Recurrent Neural Network Rachel Hu and Zhi Zhang Amazon AI d2l.ai Outline Dependent Random Variables Text Preprocessing Language Modeling Recurrent Neural Networks (RNN) LSTM Bidirectional RNN Deep RNN d2l.ai

798 views • 55 slides

Single Layer Recurrent Network Bidirectional Symmetric Connection Binary /

Hopfield Network Single Layer Recurrent Network Bidirectional Symmetric Connection Binary / Continuous Units Associative Memory Optimization Problem Hopfield Model Discrete Case Recurrent neural network that uses

288 views • 8 slides

! Single Layer Recurrent Network ! Bidirectional Symmetric Connection ! Binary /

Hopfield Network ! Single Layer Recurrent Network ! Bidirectional Symmetric Connection ! Binary / Continuous Units ! Associative Memory ! Optimization Problem Hopfield Model Discrete Case Recurrent neural network that

733 views • 51 slides

RNN Input Layer RNN Hidden Layer RNN h t-1 h t x t (Picture adapted from Andrej

Outline Recurrent Neural Networks (RNNs) Neural Machine Transla/on NMT basics (Sutskever et al., 2014) ABenCon mechanism (Bahdanau et al., 2015) Marcello Federico 2016 Recurrent Neural Networks (RNNs) Based on slides kindly provided

437 views • 24 slides

Understanding Hidden Memories of Recurrent Neural Networks Yao Ming , Shaozu Cao, Ruixiang Zhang,

Understanding Hidden Memories of Recurrent Neural Networks Yao Ming , Shaozu Cao, Ruixiang Zhang, Zhen Li, Yuanzhe Chen, Yangqiu Song, Huamin Qu. THE HONG KONG UNIVERSITY OF SCIENCE AND TECHNOLOGY What is a Recurrent Neural Network? H K U S T

397 views • 37 slides

Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling Authors: Junyoung

Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling Authors: Junyoung Chung, Caglar Gulcehre, KyungHyun Cho and Yoshua Bengio Presenter: Yu-Wei Lin Background: Recurrent Neural Network Traditional RNNs encounter

229 views • 18 slides

CONVOLUTIONAL AND RECURRENT NEURAL NETWORKS Neural networks Fully connected networks

CONVOLUTIONAL AND RECURRENT NEURAL NETWORKS Neural networks Fully connected networks Neuron Non-linearity Softmax layer DNN training Loss function and regularization SGD and backprop Learning rate Overfitting

1.8k views • 86 slides

Recurrent Neural Network Agenda Recurrent Neural Network

Recurrent Neural Network Agenda Recurrent Neural Network NER sequence tagging RNN

670 views • 46 slides