Large-scale deployment of statistical machine translation Example - PDF document

10/26/2008 Large-scale deployment of statistical machine translation Example Microsoft Chris.Wendt@microsoft.com Microsoft Research – Machine Translation Agenda • Microsoft MT engine basics • Architecture and design for scale • Translator in Practice • Microsoft internal use: Human Translation and Raw Publishing 1

10/26/2008 Time Line • Microsoft Research is founded, with NLP as one of its first research areas • NLP team is active in rule-based parsing and grammar checking 1991 •Grammar Checker in Word ‘97 1996 • Work on Machine Translation begins 1999 • V1: First public visibility with the Microsoft Knowledge Base • Example-based system: V1 of Microsoft Translator 2003 • V2: Switch to Treelet systems for all from English language pairs • Treelet system consitutes V2 of Microsoft Translator 2005 • First consumer availability at http://translator.live.com in 2007 2007 • Mixed Systran and Microsoft Translator V2 deployment • Adding a phrasal systems for all to English language pairs 2008 •http://translator.live.com powered exclusively by Microsoft‘s own systems Microsoft’s Statistical MT Engine Languages with source parser Source language Syntactic tree based decoder parser Rule-based post HTML handling processing Sentence breaking Case restoration Source language Surface string based decoder word breaker Distance and Contextual Syntactic Other source languages word-based translation reordering reordering model model Target Syntactic word Models language insertion and model deletion model 2

10/26/2008 Training Architecture Parallel Source language Data parsing Model Discrim. Train weights model weights Treelet + Source/Target Word alignment Syntactic structure word breaking extraction Target language monolingual data Language Surface Phrase table Treelet table Syntactic models model reordering extraction extraction training training training Case Target Distance and Contextual Syntactic Syntactic word restoration language word-based translation reordering insertion and model model reordering models model deletion model Runtime Architecture Translator #1 Front Door Machine #1 Model Server #1 Translator #2 User Interface Traffic Distribution Sentence Breaking … .. … .. Translator #3 Internet … .. Front Door Machine #n Model Server #n Translator #n-1 User Interface Sentence Breaking Translator #n Watchdog #1 Watchdog #1 Monitor, reset, restart Monitor, reset, restart 3

10/26/2008 Front Door • Microsoft Internet Information Server • Landing Page – HTTP interface for Bilingual Viewer • Fetches web page, sentence & html breaking, creates marked up version • Sends page to client, asynchronously fills translation requests • Distributor – SOAP API – Distributes sentences to multiple leaves – In memory cache of sentence translations Automatic evaluation: BLEU • A fully automated MT evaluation metric – Modified N-gram precision, comparing a test sentence to reference sentences • Automatic and cheap: runs daily and for every check- in • Standard in the MT community – Immediate, simple to administer – Shown to correlate with human judgments • Warning: Does not compare between engines or between languages. 4

10/26/2008 Human evaluations • 3 to 5 independent human evaluators are asked to rank translation quality for 250 sentences on a scale of 1 to 4 – Comparing to human translated sentence – No source language knowledge required 4 = Ideal Grammatically correct, all information included 3 = Acceptable Not perfect, but definitely comprehensible, and with accurate transfer of all important information 2 = Possibly Acceptable May be interpretable given context/time, some information transferred accurately 1 = Unacceptable Absolutely not comprehensible and/or little or not information transferred accurately • Each sentence is evaluated by all raters, and scores are averaged • Relative evaluations – Track progress against ourselves and a competitor 9 Language pairs on translator.live.com en_es other The fact to note in this distribution is en_es 18% the relative popularity of the en_de English>German language pair de_en 2% en_pt among consumers, in contrast to the en_ko lack of popularity for this language 3% en_zh-chs en_zh-cht pair among the technical audience. en_fr 3% en_ja es_en 3% pt_en en_de en_ar 15% 3% en_it en_it en_ar 4% en_ja pt_en en_zh-cht 4% en_ko es_en en_pt 6% de_en 13% en_fr en_zh-chs 6% 7% 5

10/26/2008 Products • Bilingual Viewer – Used by Live Search results page • Translator landing page • Toolbar Translator Button • Translator Add-in for 3 rd party pages • Internet Explorer 8 accelerator • Community built Firefox version • Translator Bot (mtbot@hotmail.com) • Office Research Pane • SOAP API for product team use • Microsoft Localization – CSS KB, MSDN Technet, Products Two ways to apply MT in a product • Post-Editing • Raw publishing – Increase human – Publish the output of the translator’s productivity MT system directly to end user – In practice: 0% to 25% – Best with bilingual UI productivity increase • Varies by content, style – Good results with IT Pro and language and Developer audience  Increasing the extent of localization 6

10/26/2008 MT with post-editing Translation memory (TM) Apply TM on Human Source Target >85% match editing Apply MT on the rest Product 1: Post-editing Results Without specific post-editor training Productivity gain/loss -23% Japanese Chinese T. 1.80% Chinese S. 11% -9.20% Spanish Productivity gain/loss -12% German -3.80% Italian French 8% -30% -20% -10% 0% 10% 20% 7

10/26/2008 Product 2: Post-Editing Results A couple of weeks later: with training Productivity gain Dutch 14.70% Czech 6.10% Danish 28.60% Productivity gain Swedish 8% Brazilian 20.00% French 14.50% 0.00% 5.00% 10.00% 15.00% 20.00% 25.00% 30.00% 35.00% Product 3: Post-editing Results With training Productivity gain/loss Brazilian -1.90% Japanese -4.05% Chinese T. 6.56% Chinese S. 22.01% Spanish 10.98% Productivity gain/loss German 29.12% Italian 33.35% French 11.89% -10.00% 0.00% 10.00% 20.00% 30.00% 40.00% 8

10/26/2008 Post-Editing: Lessons Learned • Training of the translator is required – Understand the peculiarities of the engine used – Always read the source sentence first – Understand when to discard the MT • “Two seconds is Too much” • Acknowledge different suitability for different style and terminology • Customize terminology per individual project – use of dictionary • Productivity gains of 5% to 25% are achievable, but investment is required Raw MT Publishing Translation memory (TM) Apply TM on Source Target 100% match Apply MT on the rest 9

10/26/2008 10

10/26/2008 History of MT in Customer Support • Since 2003 CSS has been actively using Machine Translation for Knowledge Base articles – Spanish was the first language deployed – Japanese went live one year later • Current Languages – 10 Languages deployed: Spanish, German, French, Italian, Japanese, Portuguese, Brazilian Portuguese, Chinese Simplified, Chinese Traditional, Arabic – 3 Languages in Testing: Korean, Turkish and Russian Microsoft Knowledge Base MT & HT distribution across languages Human translated, or Traffic to the knowledge base is originally authored in fairly unevenly distributed. By language Language % targeting human translation to the high page view articles, 80% of the English 235,425 100% Japanese total page views are for Japanese 70,684 27% human translated articles. Even in Arabic 54% of the page French 35,310 14% views end up on human quality German 30,459 12% articles. Spanish 16,980 7% Italian 14,401 6% Chinese (Simplified) 12,873 5% Chinese (Traditional) 10,372 4% Portuguese (Brazil) 10,205 4% Portuguese (Iberian) 7,129 3% Arabic 2,152 1% 11

10/26/2008 Customer Feedback: KB Inline Survey Knowledge Base – average resolve rate of human translated vs. machine translated articles 24.1% Spanish 29.2% 28.7% Portuguese (Brazil) 28.7% 23.3% Portuguese 27.6% 17.8% Japanese 27.8% 26.5% Italian 33.3% 18.7% German 25.0% 20.9% French 22.5% 29.0% Chinese (Traditional) 35.4% 32.6% Chinese (Simplified) 35.3% 25.4% Arabic 31.8% 25.5% English Machine Translation Human Translation 12

10/26/2008 Global English • Support started rewriting source language to account for MT in 2003 (6 months after Spanish MT was deployed) • Retrained the writers to write with global audience and MT in mind. • Top five rules to make source language content suitable for MT: 1. Use Standard English writing style 2. Use correct punctuation – especially the following: – Missing punctuation causing incorrect sentence break – Hyphens – Commas 3. Eliminate long sentences 4. Use capitalization correctly 5. Use correct spelling Impact of Global English Resolve rate of articles authored to standard guidelines 25% MT Languages Combined 22% 40% ZH-TW - Chinese Traditional 27% 31% ZH-CN - Chinese Simplified 26% 36% PT-BR - Portuguese Brazil 22% 27% PT - Portuguese 25% 20% JA - Japanese 18% 34% IT - Italian 24% 24% FR-French 20% 22% ES - Spanish 23% 19% DE - German 19% 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% % Yes - Global English % Yes - Old Article 13