Automatic Estimation of Simultaneous Interpreter Performance Craig - PowerPoint PPT Presentation

Automatic Estimation of Simultaneous Interpreter Performance Craig Stewart 1 , Nikolai Vogler 1 , Junjie Hu 1 , Jordan Boyd-Graber 2 , Graham Neubig 1 1 Language Technologies Institute, Carnegie Mellon University 2 CS, iSchool, UMIACS, LSC, University of Maryland 1

Simultaneous Interpretation (SI) Translation of the spoken word in real time 2

Computer Assisted Interpretation (CAI) 3

Computer Assisted Interpretation (CAI) How do we ensure maximum utility with minimum distraction ? 4

Estimating Interpreter Performance Don’t offer help when they don’t need it! Estimate how well the interpreter is doing 5

Quality Estimation We already do this in Machine Translation ! Can we apply it to Simultaneous Interpretation ? 6

Quality Estimation QuEst++ is an existing framework for QE (Specia et al., 2015) 7

Quality Estimation QuEst++ is an existing framework for QE (Specia et al., 2015) Source/Target Sentence Pairs 8

Quality Estimation QuEst++ is an existing framework for QE (Specia et al., 2015) Source/Target Sentence Pairs Extract Features 9

Quality Estimation QuEst++ is an existing framework for QE (Specia et al., 2015) Source/Target Reference Sentence Pairs Translations Extract Features 10

Quality Estimation QuEst++ is an existing framework for QE (Specia et al., 2015) Source/Target Reference Sentence Pairs Translations Apply Extract Evaluation Features Metric 11

Quality Estimation QuEst++ is an existing framework for QE (Specia et al., 2015) Source/Target Reference Sentence Pairs Translations Apply Learn Extract Evaluation Model Features Metric 12

Quality Estimation QuEst++ is an existing framework for QE (Specia et al., 2015) Source/Target Reference Sentence Pairs Translations Apply Learn Extract Evaluation Model Features Metric Score Predictions 13

Method 14

Method Source/Target Reference Sentence Pairs Translations Apply Learn Extract Evaluation Model Features Metric Score Predictions 15

Method Source speaker/ Reference Interpreter transcripts Translations Apply Learn Extract Evaluation Model Features Metric Score Predictions 16

Method Source speaker/ Reference Interpreter transcripts Translations English-Japanese Apply Learn (3 interpreters) Evaluation Model English-French Metric English-Italian Score Predictions 17

Method Source speaker/ Reference Interpreter transcripts Translations QuEst++ baseline Apply features Learn Evaluation Model Metric Score Predictions 18

Method Source speaker/ Reference Interpreter transcripts Translations QuEst++ baseline Apply features Learn Evaluation Model Metric Features tailored to interpretation Score Predictions 19

Method Source speaker/ Reference Interpreter transcripts Translations QuEst++ baseline Apply features Evaluation Learn Metric Model Features tailored to interpretation (METEOR) Score Predictions 20

Method Source speaker/ Reference Interpreter transcripts Translations Learn QuEst++ baseline Apply Model features Evaluation Metric (Support Features tailored Vector to interpretation (METEOR) Regression) Score Predictions 21

Method Source speaker/ Reference Interpreter transcripts Translations Learn QuEst++ baseline Apply Model features Evaluation Metric (Support Features tailored Vector to interpretation (METEOR) Regression) Test using 10-fold cross-validation 22

Baseline Features ● Number of words ● Average word length ● Language model probability ● N-gram frequency ● Punctuation 23

Features of interpretation SOURCE: “Will the Parliament grant President Dilma Rousseff, on the very first occasion after her groundbaking groundbreaking election and for no sound formal reason, the kind of debate that we usually reserve for people like Mugabe? So, I ask you to remove Brazil from the agenda of the urgencies.” ( 48 words ) INTERP: “Ehm il Parlamento... dopo le elezioni... darem- dar spazio a un dibattito sul ehm sul caso per esempio del presidente Mugabe invece di mettere il Brasile all’ordine del giorno?” ( 27 words ) GLOSS: “Ehm the Parliament... after the elections... we’ll gi- will give way to a debate on the ehm on the case for example of President Mugabe instead of putting Brazil on the agenda?” 24

Features of interpretation SOURCE: “Will the Parliament grant President Dilma Rousseff, on the very first occasion after her groundbaking groundbreaking election and for no sound formal reason, the kind of debate that we usually reserve for people like Mugabe? So, I ask you to remove Brazil from the agenda of the urgencies.” ( 48 words ) INTERP: “Ehm il Parlamento... dopo le elezioni... darem- dar spazio a un dibattito sul ehm sul caso per esempio del presidente Mugabe invece di mettere il Brasile all’ordine del giorno?” ( 27 words ) GLOSS: “Ehm the Parliament... after the elections... we’ll gi- will give way to a debate on the ehm on the case for example of President Mugabe instead of putting Brazil on the agenda?” 25

Features of interpretation SOURCE: “Will the Parliament grant President Dilma Rousseff, on the very first occasion after her groundbaking groundbreaking election and for no sound formal reason, the kind of debate that we usually reserve for people like Mugabe? So, I ask you to remove Brazil from the agenda of the urgencies.” ( 48 words ) INTERP: “ Ehm il Parlamento... dopo le elezioni... darem- dar spazio a un dibattito sul ehm sul caso per esempio del presidente Mugabe invece di mettere il Brasile all’ordine del giorno?” ( 27 words ) GLOSS: “Ehm the Parliament... after the elections... we’ll gi- will give way to a debate on the ehm on the case for example of President Mugabe instead of putting Brazil on the agenda?” 26

Features of interpretation SOURCE: “Will the Parliament grant President Dilma Rousseff, on the very first occasion after her groundbaking groundbreaking election and for no sound formal reason, the kind of debate that we usually reserve for people like Mugabe? So, I ask you to remove Brazil from the agenda of the urgencies.” ( 48 words ) INTERP: “ Ehm il Parlamento ... dopo le elezioni ... darem- dar spazio a un dibattito sul ehm sul caso per esempio del presidente Mugabe invece di mettere il Brasile all’ordine del giorno?” ( 27 words ) GLOSS: “Ehm the Parliament... after the elections... we’ll gi- will give way to a debate on the ehm on the case for example of President Mugabe instead of putting Brazil on the agenda?” 27

Features of interpretation SOURCE: “Will the Parliament grant President Dilma Rousseff, on the very first occasion after her groundbaking groundbreaking election and for no sound formal reason, the kind of debate that we usually reserve for people like Mugabe? So, I ask you to remove Brazil from the agenda of the urgencies.” ( 48 words ) INTERP: “ Ehm il Parlamento ... dopo le elezioni ... darem- dar spazio a un dibattito sul ehm sul caso per esempio del presidente Mugabe invece di mettere il Brasile all’ordine del giorno?” ( 27 words ) GLOSS: “Ehm the Parliament... after the elections... we’ll gi- will give way to a debate on the ehm on the case for example of President Mugabe instead of putting Brazil on the agenda?” 28

SI Model Features ● Pauses/hesitations/incomplete words ● Non-specific words - is the interpreter avoiding specific terminology? ● Cognates/loan words - if a word is almost identical in both languages an interpreter shouldn’t struggle with it (unless it’s a ‘false friend’!) 29

Results 30

Results 31

Analysis SOURCE: “Will the Parliament grant President Dilma Rousseff, on the very first occasion after her groundbaking groundbreaking election and for no sound formal reason, the kind of debate that we usually reserve for people like Mugabe? So, I ask you to remove Brazil from the agenda of the urgencies.” ( 48 words ) INTERP: “Ehm il Parlamento... dopo le elezioni... darem- dar spazio a un dibattito sul ehm sul caso per esempio del presidente Mugabe invece di mettere il Brasile all’ordine del giorno?” ( 27 words ) ACTUAL METEOR: 0.0789 BASELINE PREDICTION: 0.1271 SI MODEL PREDICTION: 0.0660 32

Future Work ● Evaluation Metric - finding a metric better aligned with the uniqueness of strategies in SI ● Live system integration - streamlining the system to provide instantaneous feedback ● ASR - evaluate the model on ASR output ● Speech model - enhance the model using prosodic speech features 33

Questions? cas1@cs.cmu.edu github/craigastewart/qe_sim_interp 34