Automatically Extracting Challenge Sets for Non-Local Phenomena in Neural Machine Translation

Proceedings of the 23rd Conference on Computational Natural Language Learning, pages 291–303 Hong Kong, China, November 3-4, 2019. c⃝2019 Association for Computational Linguistics 291 Automatically Extracting Challenge Sets for Non-local Phenomena in Neural Machine Translation Leshem Choshen1 and Omri Abend1,2 1School of Computer Science and Engineering, 2 Department of Cognitive Sciences The Hebrew University of Jerusalem leshem.choshen@mail.huji.ac.il, oabend@cs.huji.ac.il Abstract We show that the state-of-the-art Transformer MT model is not biased towards monotonic reordering (unlike previous recurrent neural network models), but that nevertheless, long- distance dependencies remain a challenge for the model. Since most dependencies are short- distance, common evaluation metrics will be little influenced by how well systems perform on them. We therefore propose an automatic approach for extracting challenge sets replete with long-distance dependencies, and argue that evaluation using this methodology pro- vides a complementary perspective on system performance. To support our claim, we com- pile challenge sets for English-German and German-English, which are much larger than any previously released challenge set for MT. The extracted sets are large enough to allow re- liable automatic evaluation, which makes the proposed approach a scalable and practical so- lution for evaluating MT performance on the long-tail of syntactic phenomena.1 1 Introduction The assumption that proximate source words are more likely to correspond to proximate target words has often been introduced as a bias (hence- forth, locality bias) into statistical MT systems (Brown et al., 1993; Koehn et al., 2003; Chiang, 2005). While reordering phenomena, abundant for some language pairs, violate this simplifying as- sumption, it has often proved to be a useful in- ductive bias in practice, especially when comple- mented with targeted techniques for addressing non-monotonic translation (e.g., Och, 2002; Chi- ang, 2005). For example, if an adjective precedes a noun in one language and modifies it syntacti- cally, it is likely that their corresponding words 1Our extracted challenge sets and codebase are found in https://github.com/borgr/auto_challenge_ sets. will appear close to each other in the translation — i.e., they may not be immediately adjacent or even in the same order in the translation, but it is unlikely that they will be arbitrarily distant from one another. In the era of Neural Machine Translation (NMT), such biases are implicitly introduced by the sequential nature of the LSTM architecture (Bahdanau et al., 2015, see §2). The influential Transformer model (Vaswani et al., 2017) replaces the sequential LSTMs with self-attention, which does not seem to possess this bias. We show that the default implementation of the Transformer does retain some bias, but that it can be relieved by using learned positional embeddings (§3). Long-distance dependencies (LDD) between words and phrases present a long-standing prob- lem for MT (Sennrich, 2016), as they are gener- ally more difficult to detect (indeed, they pose an ongoing challenge for parsing as well (Xu et al., 2009)), and often result in non-monotonic transla- tion if the target differs from the source in terms of its word order and lexicalization patterns. The Transformer’s indifference to the absolute position of …

Content truncated to 500 words