How to Speak a Language without Knowing It Xing Shi, Kevin Knight 1 - PowerPoint PPT Presentation

How to Speak a Language without Knowing It Xing Shi, Kevin Knight 1 Heng Ji 2 1 Information Sciences Institute Computer Science Department University of Southern California { xingshi, knight } @isi.edu 2 Computer Science Department Rensselaer Polytechnic Institute Troy, NY 12180, USA jih@rpi.edu June 24, 2014 Shi, X., Knight, K. and Ji, H. (USC & RPI) How to Speak a Language June 24, 2014 1 / 31

Overview Introduction 1 Data 2 Evaluation 3 Model 4 Training 5 Phoneme-based model Phoneme-phrase-based model Word-based model Hybrid training/decoding Experiments 6 Conclusion and Future work 7 Shi, X., Knight, K. and Ji, H. (USC & RPI) How to Speak a Language June 24, 2014 2 / 31

Introduction Can people speak a language they don’t know ? Shi, X., Knight, K. and Ji, H. (USC & RPI) How to Speak a Language June 24, 2014 3 / 31

Yes, use a phrasebook Shi, X., Knight, K. and Ji, H. (USC & RPI) How to Speak a Language June 24, 2014 4 / 31

Yes, use a phrasebook Shi, X., Knight, K. and Ji, H. (USC & RPI) How to Speak a Language June 24, 2014 5 / 31

Yes, use a phrasebook Shi, X., Knight, K. and Ji, H. (USC & RPI) How to Speak a Language June 24, 2014 6 / 31

Yes, use a phrasebook What if we want to say something beyond the phrasebook ? Shi, X., Knight, K. and Ji, H. (USC & RPI) How to Speak a Language June 24, 2014 7 / 31

Or, a speech-to-speech translator from: proto-knowledge.blogspot.com However, direct Human interactivity is much more fun ! Shi, X., Knight, K. and Ji, H. (USC & RPI) How to Speak a Language June 24, 2014 8 / 31

Our solution Easily pronounceable Both input T(S) and output T’(S) are in speaker’s language. Understandable by listener T’(S) sounds like T(F). T(F) and T(S) has the same meaning. Shi, X., Knight, K. and Ji, H. (USC & RPI) How to Speak a Language June 24, 2014 9 / 31

Our solution Demo Shi, X., Knight, K. and Ji, H. (USC & RPI) How to Speak a Language June 24, 2014 10 / 31

Our solution Shi, X., Knight, K. and Ji, H. (USC & RPI) How to Speak a Language June 24, 2014 11 / 31

Data A collection of 1312 < Chinese, English, Chinglish > phrasebook tuples. 1 1182 for training, 65 for development and 65 for test. 已 经 八 点 了 Chinese English It’s eight o’clock now 意 思 埃 特 额 克 劳 克 闹 (yi si ai te e ke lao ke nao) Chinglish 这 件 衬衫 又 时 髦 又 便 宜 Chinese English this shirt is very stylish and not very expensive 迪 思 舍 特 意 思 危 锐 思 掉 利 失 安 的 闹 特 危 锐 伊 克 思 班 西 五 Chinglish 1 Dataset at http://www.isi.edu/natural-language/mt/chinglish-data.txt Shi, X., Knight, K. and Ji, H. (USC & RPI) How to Speak a Language June 24, 2014 12 / 31

Data Frequency Rank Chinese Chinglish 1 de si 2 shi te 3 yi de 4 ji yi 5 zhi fu Table : Top 5 frequent syllables in Chinese and Chinglish Shi, X., Knight, K. and Ji, H. (USC & RPI) How to Speak a Language June 24, 2014 13 / 31

Evaluation Shi, X., Knight, K. and Ji, H. (USC & RPI) How to Speak a Language June 24, 2014 14 / 31

Evaluation Shi, X., Knight, K. and Ji, H. (USC & RPI) How to Speak a Language June 24, 2014 15 / 31

Evaluation Shi, X., Knight, K. and Ji, H. (USC & RPI) How to Speak a Language June 24, 2014 16 / 31

Model: Cascade FSTs Chinese 谢谢 你 translate.google.com MT Eword Thank you CMU Pron Dict FST A (Weide,2007) Epron TH EY N K Y UW wFST B wFST E Pinyin-split s an k e y ou FST C Deterministic Rules san ke you Pinyin FST D Pron Dict Chinglish 三可由 Shi, X., Knight, K. and Ji, H. (USC & RPI) How to Speak a Language June 24, 2014 17 / 31

Model: Cascade FSTs Chinese 谢谢 你 translate.google.com MT Eword Thank you CMU Pron Dict FST A (Weide,2007) Epron TH EY N K Y UW wFST B Need to learn from data wFST E Pinyin-split s an k e y ou FST C Deterministic Rules san ke you Pinyin FST D Pron Dict Chinglish 三可由 Shi, X., Knight, K. and Ji, H. (USC & RPI) How to Speak a Language June 24, 2014 18 / 31

Phoneme-based model Construct < Epron, Pinyin-split > training pairs. Chinese Mapping schema: 1-to-1, 1-to-2 and 2-to-1. MT g r ae n Eword FST A g e r uan Epron EM to learn parameters in wFST B, e.g. wFST E wFST B P (g e | g ). Pinyin-split Viterbi alignments: FST C grand Pinyin g r ae n d FST D g e r uan d e 哥 软 的 Chinglish Shi, X., Knight, K. and Ji, H. (USC & RPI) How to Speak a Language June 24, 2014 19 / 31

Phoneme-based model labeled Epron Pinyin-split P ( p | e ) d d 0.46 d e 0.40 d i 0.06 s 0.01 ao r u 0.26 o 0.13 ao 0.06 ou 0.01 Table : Learned translation tables for the phoneme based model Shi, X., Knight, K. and Ji, H. (USC & RPI) How to Speak a Language June 24, 2014 20 / 31

Phoneme-based model Chinese MT Alignment using phoneme-based model is fine. Eword When decoding test data, choices of target FST A phonemes are context sensitive. Epron Decoding “grandmother”: wFST E wFST B g r ae n d m ah dh er g e r an d e m u e d e Pinyin-split FST C reference Pinyin-split sequence: Pinyin g e r uan d e m a d e FST D Chinglish Shi, X., Knight, K. and Ji, H. (USC & RPI) How to Speak a Language June 24, 2014 21 / 31

Phoneme-phrase-based model Intuition: model the substitution of longer sequences 2 . Viterbi alignment using Phoneme-based model: g r ae n d m ah dh er g e r uan d e m a d e Extract phoneme phrase pairs: g → g e g r → g e r ... r → r r ae n → r uan ... 2 (Koehn et al., 2003) Shi, X., Knight, K. and Ji, H. (USC & RPI) How to Speak a Language June 24, 2014 22 / 31

Word-based Model Chinese Construct < Eword,Pinyin > training pairs. MT Mapping schema: 1-to-[1,7]. Eword EM to learn parameters in wFST E, i.e. P (nai te | night ). FST A Viterbi alignments: Epron wake up wFST E wFST B wei ke a pu Pinyin-split Error happen due to sparsity: “tips” and “ti pu FST C si” only appear once. Pinyin accept tips a ke sha pu te ti pu si FST D Chinglish Shi, X., Knight, K. and Ji, H. (USC & RPI) How to Speak a Language June 24, 2014 23 / 31

Hybrid training Intuition: Combine two models during training Chinese phrase. MT Use phoneme-based model to help word-based model: Eword FST A Epron wFST B wFST E Pinyin-split FST C Pinyin Errors are fixed: FST D accept tips a ke sha pu te ti pu si Chinglish Shi, X., Knight, K. and Ji, H. (USC & RPI) How to Speak a Language June 24, 2014 24 / 31

Hybrid decoding Intuition: Combine two models during decoding phrase. Chinese MT Eword seen unseen FST A word word Epron wFST B wFST E Pinyin-split FST C Pinyin FST D Chinglish Shi, X., Knight, K. and Ji, H. (USC & RPI) How to Speak a Language June 24, 2014 25 / 31

Experiments: Sample system output 等等 我 Chinese Reference English wait for me 唯 特 佛 密 (wei te fo mi) Reference Chinglish 位 忒 佛 密 (wei te fo mi) Hybrid Chinglish Human-dictated English wait for me ASR English wait for me 年 夜 饭 都 要 吃 些什么 Chinese Reference English what do you have for the Reunion dinner 沃 特 杜 又 海 夫 佛 则 锐 又 尼 恩 低 呢 Reference Chinglish 我 忒 度 优 嗨 佛 佛 得 瑞 优你 恩 低 呢 Hybrid Chinglish Human-dictated English what do you have for the reunion dinner ASR English what do you high for 43 Union Cena Shi, X., Knight, K. and Ji, H. (USC & RPI) How to Speak a Language June 24, 2014 26 / 31

Experiments: English-to-Pinyin decoding accuracy Error Rate Model Coverage Error Rate on covered text Word based 29/65 0.042 0.664 Word-based hybrid training 29/65 0.029 0.659 Phoneme based 63/65 0.583 0.611 Phoneme-phrase based 63/65 0.136 0.194 Hybrid training/decoding 63/65 0.115 0.175 Shi, X., Knight, K. and Ji, H. (USC & RPI) How to Speak a Language June 24, 2014 27 / 31

Experiments: Human Dictation Accuracy Error Rate Model vs. reference English Dictation from Reference Chinglish 0.477 Phoneme based 0.696 Hybrid training and decoding 0.496 Shi, X., Knight, K. and Ji, H. (USC & RPI) How to Speak a Language June 24, 2014 28 / 31

Experiments: No Human in the Loop Error Rate Model vs. reference English Word based 0.925 Word-based hybrid training 0.925 Phoneme based 0.937 Phoneme-phrase based 0.896 Hybrid training and decoding 0.898 Shi, X., Knight, K. and Ji, H. (USC & RPI) How to Speak a Language June 24, 2014 29 / 31

Conclusion & Future work Conclusion Goal: Help people speak foreign languages Provide native phonetic spellings that approximate the sounds of foreign phrases Use a cascade of FSTs Improve the model by adding phrases and combining models in both training and decoding phase For future: More Language Pairs Shi, X., Knight, K. and Ji, H. (USC & RPI) How to Speak a Language June 24, 2014 30 / 31

Thank you! & QA Demo: http:\\cage.isi.edu:8080 Shi, X., Knight, K. and Ji, H. (USC & RPI) How to Speak a Language June 24, 2014 31 / 31