Vector Quantized Neural Networks for Acoustic Unit Discovery - PowerPoint PPT Presentation

Vector Quantized Neural Networks for Acoustic Unit Discovery Benjamin van Niekerk, Leanne Nortje, Herman Kamper

The Generative Factors of Speech HH / Y / UW / M / ER HUMOUR Content: Prosody: Timbre: ● Discrete phonetic units. ● Rhythm ● Quality of a particular voice. ● ≅ 44 phonemes in English. ● Intonation ● Characterized by frequency ● Stresses spectrum.

The Generative Factors of Speech HH / Y / UW / M / ER Content: Prosody: Timbre: ● Discrete phonetic units. ● Rhythm ● Quality of a particular voice. ● ≅ 44 phonemes in English. ● Intonation ● Characterized by frequency ● Stresses spectrum.

The Generative Factors of Speech Content: Prosody: Timbre: ● Discrete phonetic units. ● Rhythm ● Quality of a particular voice. ● ≅ 44 phonemes in English. ● Intonation ● Characterized by frequency ● Stresses spectrum.

What is Acoustic Unit Discovery? The goal is to learn discrete representations of speech that separate phonetic content from the other factors. …all without any labels or annotations!

What is Acoustic Unit Discovery? The goal is to learn discrete representations of speech that separate phonetic content from the other factors. …all without any labels or annotations! Encoder

Applications Bootstrap training of low-resource speech systems: Automatic speech recognition Text-to-speech Non-parallel voice conversion

But, how do we learn discrete representations using neural networks?

But, how do we learn discrete representations using neural networks? A. van den Oord, and O. Vinyals. “Neural discrete representation learning.” Advances in Neural Information Processing Systems . 2017.

Vector Quantization Layer Codebook

Vector Quantization Layer Codebook Encoder

Our contribution: we propose and compare two models for acoustic unit discovery in the ZeroSpeech 2020 Challenge . A Vector-Quantized Variational A combination of Vector-Quantization and 1. 2. Autoencoder (VQ-VAE) Contrastive Predictive Coding (VQ-CPC) VQ layer Encoder Decoder Inspired by: J. Chorowski, et al. “Unsupervised speech representation learning using wavenet autoencoders.” IEEE/ACM transactions on audio, speech, and language processing. 2019.

Our contribution: we propose and compare two models for acoustic unit discovery in the ZeroSpeech 2020 Challenge . A Vector-Quantized Variational A combination of Vector-Quantization and 1. 2. Autoencoder (VQ-VAE) Contrastive Predictive Coding (VQ-CPC) VQ layer Encoder Decoder Inspired by: J. Chorowski, et al. “Unsupervised speech representation learning using wavenet autoencoders.” Inspired by: A. van den Oord, et al. “Representation Learning with Contrastive Predictive Coding.” 2018. IEEE/ACM transactions on audio, speech, and language processing. 2019.

Vector-Quantized Variational Autoencoder VQ layer Encoder Decoder

Vector-Quantized Variational Autoencoder VQ layer Encoder Decoder minimize reconstruction error

Vector-Quantized Variational Autoencoder Information bottleneck VQ layer Encoder Decoder

Vector-Quantized Variational Autoencoder Information bottleneck VQ layer Encoder Decoder Speaker

Vector-Quantized Variational Autoencoder Information bottleneck VQ layer Encoder Decoder Powerful autoregressive Speaker model

Vector-Quantized Contrastive Predictive Coding Prediction Input

Vector-Quantized Contrastive Predictive Coding Encoder Input

Vector-Quantized Contrastive Predictive Coding VQ layer Encoder Input

Vector-Quantized Contrastive Predictive Coding Context model VQ layer Encoder Input

Vector-Quantized Contrastive Predictive Coding Predictions Context model VQ layer Encoder Input

Vector-Quantized Contrastive Predictive Coding Context vector

Vector-Quantized Contrastive Predictive Coding Positive example Context vector

Vector-Quantized Contrastive Predictive Coding Positive example Context vector Negative examples

Evaluation - Voice Conversion Evaluation Metrics: VQ layer Encoder Decoder ● Speaker similarity (1-5 scale). ● Intelligibility (character error rate). ● Mean opinion score (1-5 scale).

Evaluation - Voice Conversion Source Converted Target Other Conversion

Evaluation - Voice Conversion

Evaluation - ABX Score Triphone A: bug Encoder

Evaluation - ABX Score Triphone A: Triphone B: bug bag Encoder Encoder

Evaluation - ABX Score Triphone A: Triphone X: Triphone B: bug bag bag Encoder Encoder Encoder

Evaluation - ABX Score

Questions?

Vector Quantized Variational Autoencoder Bottleneck Encoder linear(64) VQ(512) ReLU Decoder batchnorm conv 3 ( 768 ) jitter(0.5) embedding ReLU concat batchnorm upsample conv 3 ( 768 ) biGRU(128) ReLU biGRU(128) batchnorm 50Hz upsample conv 4 stride 2 ( 768 ) ReLU GRU(896) batchnorm linear(256) conv 3 ( 768 ) embedding ReLU ReLU linear(256) batchnorm ReLU conv 3 ( 768 ) 100Hz sample mu-law softmax log-Mel spec speaker

Vector Quantized Neural Networks for Acoustic Unit Discovery - PowerPoint PPT Presentation

Vector Quantized Neural Networks for Acoustic Unit Discovery Benjamin van Niekerk, Leanne Nortje, Herman Kamper The Generative Factors of Speech HH / Y / UW / M / ER HUMOUR Content: Prosody: Timbre: Discrete phonetic units. Rhythm

Unsupervised acoustic unit discovery for speech synthesis using discrete latent-variable neural

DCASE 2016 CONVOLUTIONAL NEURAL NETWORKS FOR ACOUSTIC SCENE CLASSIFICATION Michele Valenti 1

Deep Neural Networks and Hidden Markov Models in i-vector-based Text-Dependent Speaker

Convolutional Neural Networks (CNNs) Recurrent Neural Networks (RNNs) L1 Scalar Processor L0

Training Neural Networks CMSC 470 Marine Carpuat Neural Networks so far Powerful non-linear

General Neural Networks Compositions of linear maps and component-wise non- linearities Neural

Outline Evolution of neurocomputing Artificial neural networks Feed forward

Acoustic Scene Classification by Ensembling Gradient Boosting Machine and Convolutional Neural

CSC421 Intro to Artificial Intelligence UNIT 32: Instance-based Learning and Neural Networks

Extremely Low-bit Convolution Optimization for Quantized Neural Network on Modern Computer

Neural Networks 0. Logistics Spring 2019 1 Neural Networks are taking over! Neural networks

Neural Networks 1. Introduction Fall 2017 Neural Networks are taking over! Neural networks

CONVOLUTIONAL AND RECURRENT NEURAL NETWORKS Neural networks Fully connected networks

Neural Networks 1. Introduction Spring 2019 1 Neural Networks are taking over! Neural

Neural Networks 1. Introduction Spring 2020 1 Neural Networks are taking over! Neural

Relaxation and Hopfield Networks Neural Networks Neural Networks - Hopfield 1 Bibliography

Bird Call Recognition with Artificial Neural Networks, Support Vector Machines, and Kernel

Identifying Important Features for Intrusion Detection Using Support Vector Machines and Neural

Neural Networks and their Application to Go Neural Networks Learning Blackjack Theory Training

(Very) Brief Introduction to Neural Networks IITP-03 Algorithms for NLP 1 / 31 Learning

How Neural Networks (NN) Biological Neuron: A . . . Can (Hopefully) Learn Artificial Neural . .

Newton Methods for Neural Networks: Gauss Newton Matrix-vector Product Chih-Jen Lin National

Outline Why model neural networks? Modeling Neural Networks A brief look at the neuron.

Introduction to Deep Neural Networks 0. Logistics Spring 2020 1 Neural Networks are taking