analysis by synthesis of speech prosody from data to
play

Analysis by Synthesis of Speech Prosody: from Data to Models. - PowerPoint PPT Presentation

Jan 05, 2023 •363 likes •1.47k views

Analysis by Synthesis of Speech Prosody: from Data to Models. Daniel Hirst Laboratoire Parole et Langage, CNRS & Universit de Provence, Aix en Provence, France With the past, present and future collaboration of: Caroline Bouzon

  1. Prosodic structure Foot Foot -pec- They ex- -ted his e- -lec- -tion Word Word Word Word Scuola Normale Superiore, Pisa 2009 March 13 06/03/10 ATILF Nancy Daniel Hirst

  2. Prosodic structure ● Narrow rhythm unit (Jassem): sequence of syllables beginning with a stressed syllable and ending at the following word boundary ● Anacrusis (Jassem): sequence of unstressed syllables not included in a narrow rhythm unit. 06/03/10 ATILF Nancy Daniel Hirst

  3. Prosodic structure Foot Foot Ana NRU Ana NRU -dic- They pre- -ted his e- -lec- -tion Word Word Word Word 06/03/10 ATILF Nancy Daniel Hirst

  4. Aix-Marsec database • SEC (Spoken English Corpus) Knowles et al. 1996 • Marsec (Machine Readable SEC) Roach et al. 1993 • Aix-Marsec Auran, Bouzon & Hirst 2004 06/03/10 ATILF Nancy Daniel Hirst

  5. SEC ● 5.5 hours of “authentic” speech ● 53 speakers, c. 55000 words 06/03/10 ATILF Nancy Daniel Hirst

  6. SEC ● 5.5 hours of “authentic” speech ● c. 55000 words, 53 speakers ● Prosodic markup:tonetic stress marks (Knowles & Williams) Scuola Normale Superiore, Pisa 2009 March 13 06/03/10 ATILF Nancy Daniel Hirst

  7. Marsec ● Tonetic stress markup > ASCII (Roach et al.) ● words aligned with signal 06/03/10 ATILF Nancy Daniel Hirst

  8. Aix-Marsec database ● Phonetic transcription ● Phonemes aligned with signal ● Prosodic structure (Praat TextGrids) ● Automatic analysis of intonation (Momel & INTSINT) ● Freely available from the authors 06/03/10 ATILF Nancy Daniel Hirst

  9. TextGrid from Aix-Marsec 06/03/10 ATILF Nancy Daniel Hirst

  10. Hypothesis ● size of whole :: compression of parts If a prosodic constituent is involved in the planning of speech rhythm we should expect the size of the constituent to have a negative effect on the duration of the phonemes which make it up. 06/03/10 ATILF Nancy Daniel Hirst

  11. Method ● Linear correlation and regression – Independent variable: size of constituent (number of phonemes) – Dependent variable: mean lengthening/compression of phonemes (Z score) z i / p = d i / p - m p s p 06/03/10 ATILF Nancy Daniel Hirst

  12. Results - 1 ● Very significant negative correlation of lengthening of phonemes (Z-score) with number of phonemes in – Word – Foot – Narrow Rhythm Unit 06/03/10 ATILF Nancy Daniel Hirst

  13. Results - 2 ● Little or no correlation of lengthening/compression of phonemes (Z-score) with number of phonemes in: – Syllable – Anacrusis 06/03/10 ATILF Nancy Daniel Hirst

  14. Interpretation ● Syllable and anacrusis have little effect on the lengthening of English phonemes ● Word, foot and narrow rhythm unit play significant role (in that order) 06/03/10 ATILF Nancy Daniel Hirst

  15. Prosodic structure Foot Foot Ana NRU Ana NRU -pec- They ex- -ted his e- -lec- -tion Word Word Word Word 06/03/10 ATILF Nancy Daniel Hirst

  16. Results - 3 ● No simple effect of stress !!! 06/03/10 ATILF Nancy Daniel Hirst

  17. Final lengthening 06/03/10 ATILF Nancy Daniel Hirst

  18. Excluding last two phonemes of intonation unit 06/03/10 ATILF Nancy Daniel Hirst

  19. Word-final lengthening? 06/03/10 ATILF Nancy Daniel Hirst

  20. Conclusions ● No compression at level of syllable (cf Jassem et al. 1978) ● Phonemes in stressed syllable have NO specific lengthening (cf Jassem 1952!) ● The solution to Klatt’s unsolved problem is the Narrow Rhythm Unit (for English) (cf Jassem 1952!!!) ● No evidence for specific word-final lengthening 06/03/10 ATILF Nancy Daniel Hirst

  21. Duration of NRU / number of phonemes in NRU 06/03/10 ATILF Nancy Daniel Hirst

  22. mean z-score of phoneme / position in NRU 06/03/10 ATILF Nancy Daniel Hirst

  23. modelling speech melody ● Perception models ● Production models ● Acoustic models 06/03/10 ATILF Nancy Daniel Hirst

  24. Raw f0 06/03/10 ATILF Nancy Daniel Hirst

  25. Raw f0 06/03/10 ATILF Nancy Daniel Hirst

  26. raw f0 06/03/10 ATILF Nancy Daniel Hirst

  27. Raw f0 06/03/10 ATILF Nancy Daniel Hirst

  28. Finnish 06/03/10 ATILF Nancy Daniel Hirst

  29. Kloker 1975 06/03/10 ATILF Nancy Daniel Hirst

  30. Gamma function: y = at b e ct 06/03/10 ATILF Nancy Daniel Hirst

  31. Hirst's law An acoustic model should not depend on which end of the table you are talking about. 06/03/10 ATILF Nancy Daniel Hirst

  32. f0 transition 06/03/10 ATILF Nancy Daniel Hirst

  33. First derivative of raw f0 But who stole Jane's bicycle? (ma'ma'ma...) 06/03/10 ATILF Nancy Daniel Hirst

  34. Quadratic spline function • Spline function ● Sequence of functions of degree n, derivatives of which up to n-1 are everywhere continuous • Quadratic spline ● Sequence of targets linked by two quadratic functions (y = ax 2 + bx +c) 06/03/10 ATILF Nancy Daniel Hirst

  35. Quadratic spline function y =h 1 +(h 2 -h 1 )(x-t 1 ) 2 y =h 2 +(h 1 -h 2 )(x-t 2 ) 2 (t k -t 1 )(t 2 -t 1 ) (t k -t 2 )(t 1 -t 2 ) 06/03/10 ATILF Nancy Daniel Hirst

  36. Quadratic spline function Il faut que je sois à Grenoble, Samedi vers quinze heures 06/03/10 ATILF Nancy Daniel Hirst

  37. Curves vs. straight lines • 't Hart 1991 2 4 2 0 0 2 0 0 5 1 9 5 1 9 5 1 9 0 1 9 0 1 8 5 1 8 5 3 1 8 0 1 8 0 1 7 5 1 7 5 1 7 0 1 7 0 1 1 6 5 1 1 6 5 2 1 6 0 1 6 0 1 5 5 1 5 5 1 5 0 1 5 0 06/03/10 ATILF Nancy Daniel Hirst

  38. Automatic Momel ● Hirst & Espesser 1993 Asymmetric quadratic modal regression • Modal • Quadratic • Asymmetric 06/03/10 ATILF Nancy Daniel Hirst

  39. Mean and Mode mode mean 06/03/10 ATILF Nancy Daniel Hirst

  40. Mean and Mode • Mean value minimising sum of squares of diferences from data • Mode value minimising number of cases more than ∆ from data Generalise to function • Linear regression function minimising sum of squares of diferences from data • Modal regression function minimising number of cases more than ∆ from data 06/03/10 ATILF Nancy Daniel Hirst

  41. Asymmetric regression • no values more than Δ above the function • Minimise number of values more than Δ below it • Here, function is f = at 2 + bt + c 06/03/10 ATILF Nancy Daniel Hirst

  42. Momel ● Hirst & Espesser 1993 06/03/10 ATILF Nancy Daniel Hirst

  43. Evaluation of Momel ● Estelle Campione, 2001 06/03/10 ATILF Nancy Daniel Hirst

  44. Improved algorithm 06/03/10 ATILF Nancy Daniel Hirst

  45. Improved algorithm 06/03/10 ATILF Nancy Daniel Hirst

  46. Momel – theory neutral? ● Theory friendly ● used for – Fujisaki model (Mixdorff) – ToBI (Maghbouleh, Wightman & Cambell, Cho (K-ToBI) – INTSINT 06/03/10 ATILF Nancy Daniel Hirst

  47. INTSINT ● An INternational Transcription System for INTonation ● Based on minimal pitch contrasts in descriptions of intonation patterns ● Used in Hirst & Di Cristo 1998 for 9 different languages – British English, Spanish, European Portuguese, Brazilian Portuguese, French, Romanian, Russian, Moroccan Arabic and Japanese ● Extension for duration and rhythm 06/03/10 ATILF Nancy Daniel Hirst

  48. Basic INTSINT ● Absolute tones T(op) M(id) B(ottom) ● Relative tones H(igher) S(ame) L(ower) ● Iterative relative tones U(pstepped) D(ownstepped) 06/03/10 ATILF Nancy Daniel Hirst

  49. 2 speaker parameters: Hirst 2005 T H S H U k D U S e M y S D L L B range 06/03/10 ATILF Nancy Daniel Hirst

  50. downdrift 2 0 0 1 5 0 1 0 0 5 0 0 M T L H L H L H B 06/03/10 ATILF Nancy Daniel Hirst

Recommend

Speech Processing 15-492/18-492 Speech Synthesis Pronunciation Letter to Sound rules Speech

Speech Processing 15-492/18-492 Speech Synthesis Pronunciation Letter to Sound rules Speech

Speech Processing 15-492/18-492 Speech Synthesis Pronunciation Letter to Sound rules Speech Synthesis Linguistic Analysis Linguistic Analysis Pronunciations Pronunciations Prosody Prosody Part of Speech Tagging

383 views • 21 slides
Speech Processing 15- -492/18 492/18- -492 492 Speech Processing 15 Speech Synthesis Prosody

Speech Processing 15- -492/18 492/18- -492 492 Speech Processing 15 Speech Synthesis Prosody

Speech Processing 15- -492/18 492/18- -492 492 Speech Processing 15 Speech Synthesis Prosody Speech Synthesis Speech Synthesis Linguistic Analysis Linguistic Analysis Pronunciations Pronunciations Prosody Prosody

420 views • 24 slides
Speech Processing 11-492/18-492 Speech Synthesis Overview Text processing Speech Synthesis

Speech Processing 11-492/18-492 Speech Synthesis Overview Text processing Speech Synthesis

Speech Processing 11-492/18-492 Speech Synthesis Overview Text processing Speech Synthesis From text to speech Text Analysis Strings of characters to words Linguistic Analysis From words to pronunciations and prosody

490 views • 25 slides
CHiVE Varying Prosody in Speech Synthesis with a Linguistically Driven Dynamic Hierarchical

CHiVE Varying Prosody in Speech Synthesis with a Linguistically Driven Dynamic Hierarchical

CHiVE Varying Prosody in Speech Synthesis with a Linguistically Driven Dynamic Hierarchical Conditional Variational Network Vincent Wan, Chun-an Chan, Tom Kenter, Jakub Vit, Rob Clark Modelling intonation in prosody A conditional variational

422 views • 6 slides
Sub-Project I Prosody, Tones and Text-To-Speech Synthesis Sin-Horng Chen (PI), Chiu-yu Tseng

Sub-Project I Prosody, Tones and Text-To-Speech Synthesis Sin-Horng Chen (PI), Chiu-yu Tseng

Sub-Project I Prosody, Tones and Text-To-Speech Synthesis Sin-Horng Chen (PI), Chiu-yu Tseng (Co-PI), Yih-Ru Wang (Co-PI), Yuan-Fu Liao (Co-PI), Lin-shan Lee, Hsin-min Wang Outline Members Theme of Sub-project I Research Roadmap

276 views • 23 slides
Cyborg speech: Deep multilingual speech synthesis for generating segmental foreign accent with

Cyborg speech: Deep multilingual speech synthesis for generating segmental foreign accent with

Cyborg speech: Deep multilingual speech synthesis for generating segmental foreign accent with natural prosody Gustav Eje Henter 1 , Jaime Lorenzo-Trueba 1 , Xin Wang 1 , Mariko Kondo 2 , Junichi Yamagishi 1,3 gustav@nii.ac.jp, jyamagis@nii.ac.jp

1.22k views • 58 slides
Modeling Prosody Pattern of Chinese Expressive Speech Application in Personalized Speech

Modeling Prosody Pattern of Chinese Expressive Speech Application in Personalized Speech

Modeling Prosody Pattern of Chinese Expressive Speech Application in Personalized Speech Conversion Zhang Zhang, Wu Zhiyong, Jia Jia, Cai Lianhong Outline Background Introduction Modeling prosody pattern of expressive speech

785 views • 21 slides
Speech Processing 15-492/18-492 Speech Synthesis Waveform generation 2 Speech Synthesis Text

Speech Processing 15-492/18-492 Speech Synthesis Waveform generation 2 Speech Synthesis Text

Speech Processing 15-492/18-492 Speech Synthesis Waveform generation 2 Speech Synthesis Text Analysis Text Analysis Chunking, tokenization, token expansion Chunking, tokenization, token expansion Linguistic Analysis

645 views • 29 slides
6-Text To Speech (TTS) Speech Synthesis Speech Synthesis Concept Speech Naturalness Phone

6-Text To Speech (TTS) Speech Synthesis Speech Synthesis Concept Speech Naturalness Phone

6-Text To Speech (TTS) Speech Synthesis Speech Synthesis Concept Speech Naturalness Phone Sequence To Speech Articulatory Approaches Concatenative Approaches HMM-based Approaches Rule-Based Approaches 1 Speech Synthesis Concept

749 views • 57 slides
Generating segment-level foreign-accented synthetic speech with natural speech prosody Gustav Eje

Generating segment-level foreign-accented synthetic speech with natural speech prosody Gustav Eje

Generating segment-level foreign-accented synthetic speech with natural speech prosody Gustav Eje HENTER, Jaime LORENZO-TRUEBA, Xin WANG, Mariko KONDO, Junichi YAMAGISHI gustav@nii.ac.jp Digital Content and Media Sciences Research Division,

751 views • 54 slides
EE E6820: Speech & Audio Processing & Recognition Lecture 6: Music analysis and synthesis

EE E6820: Speech & Audio Processing & Recognition Lecture 6: Music analysis and synthesis

EE E6820: Speech & Audio Processing & Recognition Lecture 6: Music analysis and synthesis 1 Music and nonspeech 2 Music synthesis techniques 3 Sinewave synthesis 4 Music analysis 5 Transcription Dan Ellis

745 views • 38 slides
for Speech Synthesis and Sensor Data Augmentation Deep Generative Neural Network Speech Text

for Speech Synthesis and Sensor Data Augmentation Deep Generative Neural Network Speech Text

Deep Generative Modeling for Speech Synthesis and Sensor Data Augmentation Deep Generative Neural Network Speech Text Praveen Narayanan Ford Motor Company PROJECT DESCRIPTION Use of DNNs increasingly prevalent as a solution for many data

893 views • 53 slides
Lecture 17: LPC speech synthesis and autocorrelation- based pitch tracking ECE 401, Signal and

Lecture 17: LPC speech synthesis and autocorrelation- based pitch tracking ECE 401, Signal and

Lecture 17: LPC speech synthesis and autocorrelation- based pitch tracking ECE 401, Signal and Image Analysis November 5, 2020 Outline The LPC-10 speech synthesis model The LPC-10 excitation model: white noise, pulse train Linear

517 views • 36 slides
Speech Processing 15-492/18-492 Speech Synthesis Overview Text processing Speech Synthesis

Speech Processing 15-492/18-492 Speech Synthesis Overview Text processing Speech Synthesis

Speech Processing 15-492/18-492 Speech Synthesis Overview Text processing Speech Synthesis From text to speech From text to speech Text Analysis Text Analysis Strings of characters to words Strings of characters to words

667 views • 25 slides
Text-to-Speech Synthesis Bernd Mbius Language Science and Technology Saarland University

Text-to-Speech Synthesis Bernd Mbius Language Science and Technology Saarland University

Text-to-Speech Synthesis Bernd Mbius Language Science and Technology Saarland University Lecture 4 June 4, 2020 Diphone Synthesis B Mbius Concatenative synthesis l Concatenative synthesis: general procedure Data-based,

611 views • 27 slides
The Future of Prosody Its about Time Dafydd Gibbon Bielefeld University Jinan University

The Future of Prosody Its about Time Dafydd Gibbon Bielefeld University Jinan University

The Future of Prosody Its about Time Dafydd Gibbon Bielefeld University Jinan University Speech Prosody 9, Pozna, 13 June 2018 TIME The Future of Prosody relevant topics? Ethics of research responsibility for the use of our

1.28k views • 91 slides
Chapter 9 Linear Predictive Analysis of Speech Signals 1 LPC

Chapter 9 Linear Predictive Analysis of Speech Signals 1 LPC

Chapter 9 Linear Predictive Analysis of Speech Signals 1 LPC Methods LPC methods are the most widely used in speech coding, speech synthesis, speech recognition, speaker recognition and verification and

869 views • 49 slides
Hidden Markov ov Model (HMM) based S Speech Synthesis using ing HTS Toolkit. Presenter: Omer

Hidden Markov ov Model (HMM) based S Speech Synthesis using ing HTS Toolkit. Presenter: Omer

Hidden Markov ov Model (HMM) based S Speech Synthesis using ing HTS Toolkit. Presenter: Omer Nawaz Research Officer (III) Speech Synthesis Overvie rview: Text to be Synthesized Natural Language Processing (NLP) (NLP) Speech

440 views • 18 slides
Modern speech synthesis and its implications for speech sciences Zofia Malisz 1 , Gustav Eje

Modern speech synthesis and its implications for speech sciences Zofia Malisz 1 , Gustav Eje

Modern speech synthesis and its implications for speech sciences Zofia Malisz 1 , Gustav Eje Henter 1 , Cassia Valentini-Botinhao 2 , Oliver Watts 2 , Jonas Beskow 1 , Joakim Gustafson 1 1 Division of Speech, Music and Hearing (TMH), KTH Royal

1.06k views • 63 slides
BBNANG243 Phonological analysis Prosody: Phrase stress, rhythm & intonation Zoltn G.

BBNANG243 Phonological analysis Prosody: Phrase stress, rhythm & intonation Zoltn G.

BBNANG243 Phonological analysis Prosody: Phrase stress, rhythm & intonation Zoltn G. Kiss, Pter Szigetvri, Mikls Trkenczy Dept. of English Linguistics, Etvs Lornd University z. g. kiss (elte|delg) analysis | prosody 1

660 views • 43 slides
Prosody-Based Unsupervised Speech Summarization with Two-Layer Mutually Reinforced Random Walk

Prosody-Based Unsupervised Speech Summarization with Two-Layer Mutually Reinforced Random Walk

Prosody-Based Unsupervised Speech Summarization with Two-Layer Mutually Reinforced Random Walk Sujay Kumar Jauhar {sjauhar, yvchen, fmetze}@cs.cmu.edu Yun-Nung (Vivian) Chen The 6th International Joint Conference on Natural Florian Metze

768 views • 38 slides
Using multimodal speech production data to evaluate articulatory animation for audiovisual speech

Using multimodal speech production data to evaluate articulatory animation for audiovisual speech

Data Registration Animation Evaluation Using multimodal speech production data to evaluate articulatory animation for audiovisual speech synthesis Ingmar Steiner Korin Richmond Slim Ouni N I V E U R S E I H T T Y O H F G R

386 views • 34 slides
Perceiving Prosody in Sinewave Speech A Sine of the Times Yasmine Sukola and Lissette

Perceiving Prosody in Sinewave Speech A Sine of the Times Yasmine Sukola and Lissette

Perceiving Prosody in Sinewave Speech A Sine of the Times Yasmine Sukola and Lissette Vizcarrondo Mentor: J. Nissenbaum Ph.D. NSF Grant No. 1659607 Formants. What are they and why are they so important? Formants are vocal tract resonances that

356 views • 24 slides
Introduction to NLP and NLG Introduction to NLP Rules or Statistics?? Lexical Analysis,

Introduction to NLP and NLG Introduction to NLP Rules or Statistics?? Lexical Analysis,

Introduction to NLP and NLG Introduction to NLP Rules or Statistics?? Lexical Analysis, Syntax Analysis, Semantic Analysis, Pragmatics Speech Processing (Phonetics, Punctuations, Prosody) Ambiguity Concepts

735 views • 13 slides

More recommend