F ROM WEBSITES TO APPS , AND NOW FROM APPS TO CHATBOTS ? A NTNIO B - PowerPoint PPT Presentation

F ROM WEBSITES TO APPS , AND NOW FROM APPS TO CHATBOTS ? A NTÓNIO B RANCO

FROM APPS TO CHATBOTS CEO of large social network in annual development conference 2 months ago Shared vision for next 2 decades: past : with advent of PCs, companies reached customers with websites currently : with smartphones, are reaching customers with apps future : will be reaching with chatbots António Branco | University of Lisbon META-FORUM 2016 | Lisbon, Jul 4-5, 2016 2

A REAL USAGE SCENARIO Provide accurate support to end-users via chat channel When moving to another linguistic market , accumulated advantage vanishes to zero?? António Branco | University of Lisbon META-FORUM 2016 | Lisbon, Jul 4-5, 2016 3

MACHINE TRANSLATION AT WORK Not necessarily, if one resorts to machine translation: But how much effective is using MT in day 1 in a new linguistic market? António Branco | University of Lisbon META-FORUM 2016 | Lisbon, Jul 4-5, 2016 4

HOW MUCH CAN MT HELP ? Cost with human resources are the lion’s share in contact centers … … MT contribution is as much effective as more calls to human operators can be spared in day 1: Probability* Avg.* EU ! BG ! CS ! NL ! DE ! PT ! ES ! calling*operator* low 42.2% 33.3% 47.4% 54.5% 30.4% 47.8% 21.5% 60.4% medium 20.5% 28.1% 30.6% 17.9% 21.9% 22.0% 15.8% 7.0% high 37.1% 37.0% 22.0% 27.5% 47.7% 30.1% 62.7% 32.7% * For the extrinsic evaluation methodology: Gaudio, Burchardt and Branco, 2016, "Evaluating Machine Translation in a Usage Scenario" , LREC2016 . António Branco | University of Lisbon META-FORUM 2016 | Lisbon, Jul 4-5, 2016 5

TAKE HOME MESSAGE When porting your MT-supported chat-based contact center to a new linguistic market, at day 1 of its operation, the overall chance of dispensing human operator intervention and thus the language specific costs that are spared by using MT are on average at least 40% and up to 60% of the costs of doing it without MT António Branco | University of Lisbon META-FORUM 2016 | Lisbon, Jul 4-5, 2016 6

WAIT, WAIT … - … 40%-60% of costs spared with which MT system by the way? - Off-the-shelf SMT - So, even with a less performant system than your QTLeap MT system? - Right António Branco | University of Lisbon META-FORUM 2016 | Lisbon, Jul 4-5, 2016 7

THANK YOU

EMULATE REAL USAGE Step 1: Review answer A (MT) without any reference: § It would clearly help me solve my problem / answer my question § It might help, but would require some thinking to understand it. § Is not helpful / I don't understand it Step2: Compare answers A and B (human reference), (re-)evaluate A selecting one of the following options: § A gives the right advice. § A gets minor points wrong. § A gets important points wrong. António Branco | University of Lisbon META-FORUM 2016 | Lisbon, Jul 4-5, 2016 9

RESULTS OF STEP 1 AND 2 António Branco | University of Lisbon META-FORUM 2016 | Lisbon, Jul 4-5, 2016 10

ESTIMATING OPERATOR INVENTION PROBABILITY Probability* EU ! BG ! CS ! NL ! DE ! PT ! ES ! Avg.* low 33.3% 47.4% 54.5% 30.4% 47.8% 21.5% 60.4% 42.2% medium 28.1% 30.6% 17.9% 21.9% 22.0% 15.8% 7.0% 20.5% high 37.0% 22.0% 27.5% 47.7% 30.1% 62.7% 32.7% 37.1% * António Branco | University of Lisbon META-FORUM 2016 | Lisbon, Jul 4-5, 2016 11

The Baseline • Baseline system – 7 language pairs: phrase-based SMT (Moses) • Two models: transla=on model, (mono, target) language model • Training – Europarl and other parallel and monolingual corpora • Tuning – MERT, 1 Ksentences in-domain data • Evalua=on – Automa=c metrics (as usual: BLEU, METEOR), 1Ksentences in- domain António Branco | University of Lisbon META-FORUM 2016 | Lisbon, Jul 4-5, 2016 12

Baseline – training seJngs António Branco | University of Lisbon META-FORUM 2016 | Lisbon, Jul 4-5, 2016 13

Baseline - datasets • Basque • 1.5 Msentences bilingual corpora (Elhuyar Founda=on, in-domain, etc.) • 2.2 M monolingual corpora • Bulgarian • 600 K bilingual (Europarl, in-domain LibreOffice, etc) • 3.4 M monolingual (+ Bulgarian Ref Corpus) • Czech • 15 M bilingual (Czech-English Parallel Corpus) • 18 M monolingual (+ Europarl, etc) • Dutch • 370 K bi & mono (Dutch Parallel Corpus, ½ in-domain ) • German • 4.5 M bi & mono (Europarl, in-domain, etc.) • Portuguese • 2 M bi & mono ( out-domain Europarl) • Spanish • 15 M bi & mono (Europarl, UN , in-domain, etc.) António Branco | University of Lisbon META-FORUM 2016 | Lisbon, Jul 4-5, 2016 14