It Is Not As Good As You Think! Evaluating Simultaneous Machine Translation on Interpretation Data

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6707–6715 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 6707 It is Not as Good as You Think! Evaluating Simultaneous Machine Translation on Interpretation Data Jinming Zhao1 Philip Arthur2 Gholamreza Haffari1 Trevor Cohn3 Ehsan Shareghi1,4 1Department of Data Science & AI, Monash University 2Oracle Digital Assistant, Oracle Corp. 4Language Technology Lab, University of Cambridge 3School of Computing and Information Systems, The University of Melbourne first.last@{monash.edu, unimelb.edu.au, oracle.com} Abstract Most existing simultaneous machine transla- tion (SiMT) systems are trained and evalu- ated on offline translation corpora. We ar- gue that SiMT systems should be trained and tested on real interpretation data. To illus- trate this argument, we propose an interpre- tation test set and conduct a realistic evalu- ation of SiMT trained on offline translations. Our results, on our test set along with 3 exist- ing smaller scale language pairs, highlight the difference of up-to 13.83 BLEU score when SiMT models are evaluated on translation vs interpretation data. In the absence of interpre- tation training data, we propose a translation- to-interpretation (T2I) style transfer method which allows converting existing offline trans- lations into interpretation-style data, leading to up-to 2.8 BLEU improvement. However, the evaluation gap remains notable, calling for constructing large-scale interpretation cor- pora better suited for evaluating and develop- ing SiMT systems. 1 1 Introduction Simultaneous interpretation (SI) is a task of translat- ing natural language in real time. SiMT systems are expected to generate interpreted text as if the text was produced by human interpreters while main- taining acceptable delay (Ma et al., 2019; Arthur et al., 2021). However, most current SiMT systems are trained and evaluated on offline translations dif- fering from real-life SI scenarios where translations are flexibly paraphrased, without compromising the source message (He et al., 2016; Paulik and Waibel, 2009). For instance, in Table 1 the interpretation sentence drops "at this point" and condenses "seri- ousness of this line of argument" to "agreement"; it delivers the source message as reliably as the offline translation. 1Our annotated test sets are available at https:// github.com/mingzi151/InterpretationData. Source: I′m Ich werde at this point an diesem Punkt refrain from darauf verzichten, einen to comment on the seriousness of this line of reasoning Kommentar zur Ernsthaftigkeit dieser Argumentationsweise abzugeben. Offline Translation: (At this point,) I will refrain from com- menting on the :::::::: seriousness::of::: this:::: line :: of ::::::: argument. Interpretation: I’m not going to comment on that:::::::: agreement. Table 1: Translation and interpretation differ in style while conveying the same source information. Prior work attempted to build interpretation cor- pora in a small scale (Tohyama and Inagaki, 2004; Shimizu et al., 2014; Bernardini et al., 2016), or constructed speech interpretation training corpora for MT tasks (Paulik and Waibel, 2010). But, very little attempt has been made on empirically quanti- fying the evaluation gap. An exception is Shimizu et al. (2013) which incorporated interpretation data in the training stage of a statistical MT system, but the lack …

Content truncated to 500 words