Automatic Evaluation and Analysis of Idioms in Neural Machine Translation

Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 3682–3700 May 2-6, 2023 ©2023 Association for Computational Linguistics Automatic Evaluation and Analysis of Idioms in Neural Machine Translation Christos Baziotis∗ University of Edinburgh c.baziotis@ed.ac.uk Prashant Mathur Amazon AI pramathu@amazon.com Eva Hasler Amazon AI ehasler@amazon.com Abstract A major open problem in neural machine trans- lation (NMT) is the translation of idiomatic ex- pressions, such as “under the weather”. The meaning of these expressions is not composed by the meaning of their constituent words, and NMT models tend to translate them literally (i.e., word-by-word), which leads to confusing and nonsensical translations. Research on id- ioms in NMT is limited and obstructed by the absence of automatic methods for quantifying these errors. In this work, first, we propose a novel metric for automatically measuring the frequency of literal translation errors without human involvement. Equipped with this metric, we present controlled translation experiments with models trained in different conditions (with/without the test-set idioms) and across a wide range of (global and targeted) metrics and test sets. We explore the role of monolin- gual pretraining and find that it yields substan- tial targeted improvements, even without ob- serving any translation examples of the test-set idioms. In our analysis, we probe the role of idiom context. We find that the randomly ini- tialized models are more local or “myopic” as they are relatively unaffected by variations of the idiom context, unlike the pretrained ones. 1 Introduction Neural machine translation (NMT; Sutskever et al. 2014; Bahdanau et al. 2015; Vaswani et al. 2017) struggles with the translation of rare multi-word expressions (MWE) (Koehn and Knowles, 2017). Non-compositional phrases, such as idioms (e.g., “piece of cake”), are one of the most challenging types of MWEs, because their meaning is figura- tive and cannot be derived from the meaning of their constituents (Nunberg et al., 1994; Liu, 2017). NMT models tend to translate these expressions literally (i.e., word-by-word), which leads to erro- neous translations. In this paper, our focus is on the ∗This work was done during an internship at Amazon. translation of idiomatic expressions, in contrast to most prior work, which is subsumed under MWEs in general (Constant et al., 2017; Cook et al., 2021). The absence of targeted and automatic evalua- tion is a major obstacle to advances in idiom transla- tion. Global metrics, such as BLEU (Papineni et al., 2002) consider the full translation, and thus, the ef- fects of idiom translation are overshadowed. Previ- ous efforts on targeted evaluation isolate the idiom translation using word alignments (Fadaee et al., 2018) or word edit distance (Zaninello and Birch, 2020). These approaches measure the accuracy of idiom translation but do not account for literal trans- lation errors. Shao et al. (2018) proposed a method for estimating the frequency of such errors, but it requires the creation of language-specific hand- crafted lists (i.e., blocklists) with words that corre- spond to literal translation errors. In this work1, we present a study of …

Content truncated to 500 words