unsupervised neural feature learning for speech using
play

Unsupervised neural feature learning for speech using weak top-down - PowerPoint PPT Presentation

Feb 26, 2023 •458 likes •1.21k views

Unsupervised neural feature learning for speech using weak top-down constraints Maties Machine Learning (MML), Oct. 2017 Herman Kamper Stellenbosch University http://www.kamperh.com/ Success in speech recognition 1 / 18 Success in speech

  1. Unsupervised neural feature learning for speech using weak top-down constraints Maties Machine Learning (MML), Oct. 2017 Herman Kamper Stellenbosch University http://www.kamperh.com/

  2. Success in speech recognition 1 / 18

  3. Success in speech recognition 1 / 18

  4. Success in speech recognition 1 / 18

  5. Success in speech recognition [Xiong et al., arXiv’16]; [Saon et al., arXiv’17] 1 / 18

  6. Success in speech recognition [Xiong et al., arXiv’16]; [Saon et al., arXiv’17] 1 / 18

  7. Success in speech recognition [Xiong et al., arXiv’16]; [Saon et al., arXiv’17] • Google Voice: English, Spanish, German, . . . , Zulu ( ∼ 50 languages) 1 / 18

  8. Success in speech recognition [Xiong et al., arXiv’16]; [Saon et al., arXiv’17] • Google Voice: English, Spanish, German, . . . , Zulu ( ∼ 50 languages) • An addiction to labels : 2000 hours transcribed speech audio; ∼ 350M/560M words text 1 / 18

  9. Success in speech recognition i had to think of some example speech since speech recognition is really cool [Xiong et al., arXiv’16]; [Saon et al., arXiv’17] • Google Voice: English, Spanish, German, . . . , Zulu ( ∼ 50 languages) • An addiction to labels : 2000 hours transcribed speech audio; ∼ 350M/560M words text 1 / 18

  10. Success in speech recognition i had to think of some example speech since speech recognition is really cool [Xiong et al., arXiv’16]; [Saon et al., arXiv’17] • Google Voice: English, Spanish, German, . . . , Zulu ( ∼ 50 languages) • An addiction to labels : 2000 hours transcribed speech audio; ∼ 350M/560M words text • But, there are around 7000 languages spoken in the world today 1 / 18

  12. Why learn without labels? 3 / 18

  13. Why learn without labels? • Get insight into human language acquisition [R¨ as¨ anen and Rasilo, ’15] 3 / 18

  14. Why learn without labels? • Get insight into human language acquisition [R¨ as¨ anen and Rasilo, ’15] • Language acquisition in robots [Roy, ’99]; [Renkens and Van hamme, ’15] 3 / 18

  15. Why learn without labels? • Get insight into human language acquisition [R¨ as¨ anen and Rasilo, ’15] • Language acquisition in robots [Roy, ’99]; [Renkens and Van hamme, ’15] • Analysis of audio for unwritten languages [Besacier et al., ’14] 3 / 18

  16. Why learn without labels? • Get insight into human language acquisition [R¨ as¨ anen and Rasilo, ’15] • Language acquisition in robots [Roy, ’99]; [Renkens and Van hamme, ’15] • Analysis of audio for unwritten languages [Besacier et al., ’14] • New insights and models for speech processing [Jansen et al., ’13] 3 / 18

  17. Unsupervised term discovery (UTD) [Park and Glass, TASLP’08] 4 / 18

  18. Unsupervised term discovery (UTD) [Park and Glass, TASLP’08] 4 / 18

  19. Unsupervised term discovery (UTD) [Park and Glass, TASLP’08] 4 / 18

  20. Unsupervised term discovery (UTD) [Park and Glass, TASLP’08] 4 / 18

  21. Example: Query-by-example search [Jansen and Van Durme, IS’12; Saeb et al., IS’17; Settle et al., IS’17] 5 / 18

  22. Example: Query-by-example search Spoken query: [Jansen and Van Durme, IS’12; Saeb et al., IS’17; Settle et al., IS’17] 5 / 18

  23. Example: Query-by-example search Spoken query: [Jansen and Van Durme, IS’12; Saeb et al., IS’17; Settle et al., IS’17] 5 / 18

  24. Example: Query-by-example search Spoken query: [Jansen and Van Durme, IS’12; Saeb et al., IS’17; Settle et al., IS’17] 5 / 18

  25. Example: Query-by-example search Spoken query: Useful speech system, not requiring any transcribed speech [Jansen and Van Durme, IS’12; Saeb et al., IS’17; Settle et al., IS’17] 5 / 18

  26. Unsupervised speech processing: Two problems 1. Unsupervised frame-level representation learning : 6 / 18

  27. Unsupervised speech processing: Two problems 1. Unsupervised frame-level representation learning : 6 / 18

  28. Unsupervised speech processing: Two problems 1. Unsupervised frame-level representation learning : f a ( · ) 6 / 18

  29. Unsupervised speech processing: Two problems 1. Unsupervised frame-level representation learning : Cool model f a ( · ) f a ( · ) 6 / 18

  30. Unsupervised speech processing: Two problems 1. Unsupervised frame-level representation learning : Cool model f a ( · ) f a ( · ) 2. Unsupervised segmentation and clustering : How do we discover meaningful units in unlabelled speech? 6 / 18

  31. Unsupervised frame-level representation learning: The Correspondence Autoencoder

  32. Unsupervised frame-level representation learning: The Correspondence Autoencoder Micha Elsner Daniel Renshaw Aren Jansen Sharon Goldwater

  33. Supervised representation learning using DNNs Output: predict phone states ay ey k v Input: speech frame(s) e.g. MFCCs, filterbanks 8 / 18

  34. Supervised representation learning using DNNs Output: predict phone states ay ey k v Phone classifier learned jointly Feature extractor f a ( · ) learned from data Input: speech frame(s) e.g. MFCCs, filterbanks 8 / 18

  35. Supervised representation learning using DNNs Output: predict phone states ay ey k v Phone classifier learned jointly Unsupervised modelling: No phone class targets to train network on Feature extractor f a ( · ) learned from data Input: speech frame(s) e.g. MFCCs, filterbanks 8 / 18

  36. Autoencoder (AE) neural network Reconstruct input Input speech frame [Badino et al., ICASSP’14] 9 / 18

  37. Autoencoder (AE) neural network Reconstruct input • Completely unsupervised • But purely bottom-up • Can we use top-down information? Input speech frame [Badino et al., ICASSP’14] 9 / 18

  38. Autoencoder (AE) neural network Reconstruct input • Completely unsupervised • But purely bottom-up • Can we use top-down information? • Idea: Unsupervised term discovery Input speech frame [Badino et al., ICASSP’14] 9 / 18

  39. Unsupervised term discovery (UTD) 10 / 18

  40. Unsupervised term discovery (UTD) Can we use these discovered word pairs to give weak top-down supervision? 10 / 18

  41. Weak top-down supervision: Align frames [Jansen et al., ICASSP’13] 11 / 18

  42. Weak top-down supervision: Align frames [Jansen et al., ICASSP’13] 11 / 18

  43. Weak top-down supervision: Align frames [Jansen et al., ICASSP’13] 11 / 18

  44. Autoencoder (AE) Reconstruct input Input speech frame 12 / 18

  45. Correspondence autoencoder (cAE) Frame from other word in pair Frame from one word 13 / 18

  46. Correspondence autoencoder (cAE) Frame from other word in pair Unsupervised feature extractor f a ( · ) Frame from one word 13 / 18

  47. Correspondence autoencoder (cAE) Frame from other word in pair Combine top-down and bottom-up information Unsupervised feature extractor f a ( · ) Frame from one word 13 / 18

  48. Correspondence autoencoder (cAE) Frame from other word in pair Play Unsupervised feature extractor Play Frame from one word 14 / 18

  49. Correspondence autoencoder (cAE) Train correspondence (1) (4) Train stacked autoencoder autoencoder (pretraining) Initialize weights Speech corpus (3) Unsupervised feature extractor (2) Unsupervised term discovery Align word pair frames [Kamper et al., ICASSP’15] 15 / 18

  50. Evaluation: Query-by-example search Spoken query: [Jansen and Van Durme, IS’12; Saeb et al., IS’17; Settle et al., IS’17] 16 / 18

  51. Evaluation: Isolated word query-by-example 0 . 5 0 . 4 Average precision 0 . 3 0 . 2 0 . 1 0 . 0 Autoencoder UBM-GMM TopUBM cAE 17 / 18

  52. Evaluation: Isolated word query-by-example 0 . 5 0 . 4 Average precision 0 . 3 0 . 2 0 . 1 0 . 0 Autoencoder UBM-GMM TopUBM cAE Extended: [Renshaw et al., IS’15] and [Yuan et al., IS’16] 17 / 18

  53. Summary and conclusion • Introduced correspondence autoencoder (cAE) for unsupervised frame-level representation learning • Uses top-down information from unsupervised term discovery system • Uses bottom-up initialization on large speech corpus • Unsupervised neural network model that combines top-down and bottom-up information results in large intrinsic improvements • Links with language acquisition research • Future: More analysis; different domains; practical search systems 18 / 18

  54. http://www.kamperh.com/ https://github.com/kamperh

  55. Evaluation of features: same-different task

  56. Evaluation of features: same-different task “apple” “pie” “grape” “apple” “apple” “like”

  57. Evaluation of features: same-different task “apple” “pie” “grape” “apple” “apple” “like” Treat as query “apple”

  58. Evaluation of features: same-different task “apple” “pie” “grape” “apple” “apple” “like” Treat as terms to search Treat as query “pie” “grape” “apple” “apple” “apple” “like”

  59. Evaluation of features: same-different task “apple” “pie” “grape” “apple” “apple” “like” “pie” “grape” “apple” “apple” “apple” “like”

  60. Evaluation of features: same-different task “apple” “pie” “grape” “apple” “apple” “like” “pie” “grape” “apple” “apple” “apple” “like”

Recommend

Part I Unsupervised Feature Learning with Convolutional Neural Networks Thomas Brox Computer

Part I Unsupervised Feature Learning with Convolutional Neural Networks Thomas Brox Computer

Part I Unsupervised Feature Learning with Convolutional Neural Networks Thomas Brox Computer Vision Group University of Freiburg, Germany Research funded by ERC Starting Grant VideoLearn and Deutsche Telekom Stiftung Thomas Brox Status quo:

489 views • 28 slides
Unsupervised neural and Bayesian models for zero-resource speech processing MIT CSAIL, 15 Nov.

Unsupervised neural and Bayesian models for zero-resource speech processing MIT CSAIL, 15 Nov.

Unsupervised neural and Bayesian models for zero-resource speech processing MIT CSAIL, 15 Nov. 2016 Herman Kamper University of Edinburgh; TTI at Chicago http://www.kamperh.com Speech recognition success 1 / 35 Speech recognition success 1

1.2k views • 88 slides
Unsupervised neural network based feature extraction using weak top-down constraints Herman Kamper

Unsupervised neural network based feature extraction using weak top-down constraints Herman Kamper

Unsupervised neural network based feature extraction using weak top-down constraints Herman Kamper 1 , 2 , Micha Elsner 3 , Aren Jansen 4 , Sharon Goldwater 2 1 CSTR and 2 ILCC, School of Informatics, University of Edinburgh, UK 3 Department of

751 views • 60 slides
Natural Language Understanding Lecture 11: Unsupervised Part-of-Speech Tagging with Neural

Natural Language Understanding Lecture 11: Unsupervised Part-of-Speech Tagging with Neural

Introduction Maximum Entropy Models as Emissions Embeddings as Emissions Natural Language Understanding Lecture 11: Unsupervised Part-of-Speech Tagging with Neural Networks Frank Keller School of Informatics University of Edinburgh

707 views • 29 slides
Neural networks: Unsupervised learning 1 Previously The supervised learning paradigm: given

Neural networks: Unsupervised learning 1 Previously The supervised learning paradigm: given

Neural networks: Unsupervised learning 1 Previously The supervised learning paradigm: given example inputs x and target outputs t learning the mapping between them the trained network is supposed to give correct response for any

1.01k views • 73 slides
Unsupervised Learning in Neural Networks Keith L. Downing The Norwegian University of Science and

Unsupervised Learning in Neural Networks Keith L. Downing The Norwegian University of Science and

Unsupervised Learning in Neural Networks Keith L. Downing The Norwegian University of Science and Technology (NTNU) Trondheim, Norway keithd@idi.ntnu.no January 18, 2014 Keith L. Downing Unsupervised Learning in Neural Networks Unsupervised

589 views • 28 slides
Unsupervised acoustic unit discovery for speech synthesis using discrete latent-variable neural

Unsupervised acoustic unit discovery for speech synthesis using discrete latent-variable neural

Unsupervised acoustic unit discovery for speech synthesis using discrete latent-variable neural networks Interspeech 2019, Graz, Austria Ryan Eloff, Andr e Nortje, Benjamin van Niekerk, Avashna Govender, Leanne Nortje, Arnu Pretorius, Elan

744 views • 38 slides
UNSUPERVISED LEARNING, CLUSTERING UNSUPERVISED LEARNING UNSUPERVISED LEARNING Supervised

UNSUPERVISED LEARNING, CLUSTERING UNSUPERVISED LEARNING UNSUPERVISED LEARNING Supervised

UNSUPERVISED LEARNING, CLUSTERING UNSUPERVISED LEARNING UNSUPERVISED LEARNING Supervised learning: X - y pairs, f(x) function approximation Unsupervised learning: only X, no y Exploring the space of X measurements,

440 views • 14 slides
3D Deep Clustering a clustering framework for unsupervised learning of 3D object feature

3D Deep Clustering a clustering framework for unsupervised learning of 3D object feature

3D Deep Clustering a clustering framework for unsupervised learning of 3D object feature descriptors LAMBDA Workshop on Retrieval and Shape analysis Alexandros Georgiou M.Sc. in Algorithms, Logic and Discrete Mathematics 1/12 Problem

570 views • 53 slides
Introduction to PCA Unsupervised Learning in R Unsupervised learning Two methods of

Introduction to PCA Unsupervised Learning in R Unsupervised learning Two methods of

UNSUPERVISED LEARNING IN R Introduction to PCA Unsupervised Learning in R Unsupervised learning Two methods of clustering finding groups of homogeneous items Next up, dimensionality reduction Find structure in features

904 views • 29 slides
Artificial Neural Networks (Part 3) Self-Organizing Feature Maps Christian Jacob CPSC 533

Artificial Neural Networks (Part 3) Self-Organizing Feature Maps Christian Jacob CPSC 533

Artificial Neural Networks (Part 3) Self-Organizing Feature Maps Christian Jacob CPSC 533 Winter 2001 Self-Organization In this chapter we consider unsupervised learning by self-organization . For these models, a correct output cannot be

498 views • 27 slides
Clustering ! Unsupervised learning: learning without a teacher ! Only features and no labels

Clustering ! Unsupervised learning: learning without a teacher ! Only features and no labels

So far in the course Supervised learning: learning with a teacher ! You had training data which was (feature, label) pairs and the goal was to learn a mapping from features to labels Clustering ! Unsupervised learning: learning without a

164 views • 12 slides
A Hybrid Deep Learning Approach For Chaotic Time Series Prediction Based On Unsupervised Feature

A Hybrid Deep Learning Approach For Chaotic Time Series Prediction Based On Unsupervised Feature

A Hybrid Deep Learning Approach For Chaotic Time Series Prediction Based On Unsupervised Feature Learning Norbert Ayine Agana Advisor: Abdollah Homaifar Autonomous Control & Information Technology Institute (ACIT), Departmenent of

379 views • 35 slides
An Unsupervised Autoregressive Model for Speech Representation Learning Yu-An Chung Wei-Ning Hsu

An Unsupervised Autoregressive Model for Speech Representation Learning Yu-An Chung Wei-Ning Hsu

An Unsupervised Autoregressive Model for Speech Representation Learning Yu-An Chung Wei-Ning Hsu Hao Tang James Glass Computer Science and Artificial Intelligence Laboratory Massachusetts Institute of Technology Cambridge, Massachusetts,

1.24k views • 13 slides
Almost Unsupervised Text to Speech and Automatic Speech Recognition Yi Ren*, Xu Tan*, Tao Qin,

Almost Unsupervised Text to Speech and Automatic Speech Recognition Yi Ren*, Xu Tan*, Tao Qin,

Almost Unsupervised Text to Speech and Automatic Speech Recognition Yi Ren*, Xu Tan*, Tao Qin, Sheng Zhao, Zhou Zhao, Tie-Yan Liu Microsoft Research Zhejiang University Motivation ASR and TTS can achieve good performance given large amount

400 views • 13 slides
Unsupervised speech processing using acoustic word embeddings Herman Kamper School of

Unsupervised speech processing using acoustic word embeddings Herman Kamper School of

Unsupervised speech processing using acoustic word embeddings Herman Kamper School of Informatics, University of Edinburgh TTI at Chicago MLSLP 2016: Spotlight invited talk Unsupervised speech processing Speech recognition applications

912 views • 68 slides
3. Feature Extraction 3.1 Feature Extraction from Speech or other types of audio like music

3. Feature Extraction 3.1 Feature Extraction from Speech or other types of audio like music

3. Feature Extraction 3.1 Feature Extraction from Speech or other types of audio like music See Schukat-Talamazzini Chapter 3 2 Goal of Feature Extraction Capture essential information about speech Be robust against background

741 views • 55 slides
Overview Neural networks and visual processing So far we have discussed unsupervised learning up

Overview Neural networks and visual processing So far we have discussed unsupervised learning up

Overview Neural networks and visual processing So far we have discussed unsupervised learning up to V1 For most technology applications (except perhaps compression), Mark van Rossum V1 description is not enough. Yet it is not clear how to

188 views • 18 slides
Unsupervised Discrete Sentence Representation Learning for Interpretable Neural Dialog Generation

Unsupervised Discrete Sentence Representation Learning for Interpretable Neural Dialog Generation

Code & Data : github.com/snakeztc/NeuralDialog-LAED Unsupervised Discrete Sentence Representation Learning for Interpretable Neural Dialog Generation Tiancheng Zhao, Kyusong Lee and Maxine Eskenazi Language Technologies Institute, Carnegie

730 views • 28 slides
Augmenting Supervised Neural Networks with Unsupervised Objectives for Large-scale Image

Augmenting Supervised Neural Networks with Unsupervised Objectives for Large-scale Image

Augmenting Supervised Neural Networks with Unsupervised Objectives for Large-scale Image Classification Yu Yuting Zh Zhang, Ki Kibok Le Lee, Ho Honglak Le Lee University of Michigan, Ann Arbor Unsupervised and supervised deep learning

753 views • 52 slides
Unsupervised Learning Gustavo Velasco-Hern andez Pattern Recognition, 2014 Gustavo

Unsupervised Learning Gustavo Velasco-Hern andez Pattern Recognition, 2014 Gustavo

Outline Introduction Clustering Feature Selection and Extraction Networks for Unsupervised Learning End Unsupervised Learning Gustavo Velasco-Hern andez Pattern Recognition, 2014 Gustavo Velasco-Hern andez Unsupervised Learning

912 views • 53 slides
Extending Unsupervised Neural Image Compression With Supervised Multitask Learning David Tellez,

Extending Unsupervised Neural Image Compression With Supervised Multitask Learning David Tellez,

Extending Unsupervised Neural Image Compression With Supervised Multitask Learning David Tellez, Diederik Hoppener, Cornelis Verhoef, Dirk Grunhagen, Pieter Nierop, Michal Drozdzal, Jeroen van der Laak, Francesco Ciompi david.tellezm@gmail.com

302 views • 17 slides
Unsupervised Learning and Inverse Problems with Deep Neural Networks J oan Bruna, Stphane

Unsupervised Learning and Inverse Problems with Deep Neural Networks J oan Bruna, Stphane

Unsupervised Learning and Inverse Problems with Deep Neural Networks J oan Bruna, Stphane Mallat, Ivan Dokmanic, Martin de Hoop cole Normale Suprieure www.di.ens.fr/data Deep Convolutional Networks ( x ) x ( u ) L j L 1 f M (

520 views • 47 slides
SuperGlue: Learning Feature Matching with Graph Neural Networks Paul-Edouard Sarlin 1 Daniel

SuperGlue: Learning Feature Matching with Graph Neural Networks Paul-Edouard Sarlin 1 Daniel

SuperGlue: Learning Feature Matching with Graph Neural Networks Paul-Edouard Sarlin 1 Daniel DeTone 2 Tomasz Malisiewicz 2 Andrew Rabinovich 2 Feature matching is ubiquitous 3D reconstruction Visual localization SLAM Place

256 views • 25 slides

More recommend