Tagged Back-translation Revisited: Why Does It Really Work?

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5990–5997 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 5990 Tagged Back-translation Revisited: Why Does It Really Work? Benjamin Marie Raphael Rubino Atsushi Fujita National Institute of Information and Communications Technology 3-5 Hikaridai, Seika-cho, Soraku-gun, Kyoto, 619-0289, Japan {bmarie, raphael.rubino, atsushi.fujita}@nict.go.jp Abstract In this paper, we show that neural ma- chine translation (NMT) systems trained on large back-translated data overfit some of the characteristics of machine-translated texts. Such NMT systems better translate human- produced translations, i.e., translationese, but may largely worsen the translation quality of original texts. Our analysis reveals that adding a simple tag to back-translations prevents this quality degradation and improves on average the overall translation quality by helping the NMT system to distinguish back-translated data from original parallel data during training. We also show that, in contrast to high-resource configurations, NMT systems trained in low- resource settings are much less vulnerable to overfit back-translations. We conclude that the back-translations in the training data should al- ways be tagged especially when the origin of the text to be translated is unknown. 1 Introduction During training, neural machine translation (NMT) can leverage a large amount of monolingual data in the target language. Among existing ways of exploiting monolingual data in NMT, the so-called back-translation of monolingual data (Sennrich et al., 2016a) is undoubtedly the most prevalent one, as it remains widely used in state-of-the-art NMT systems (Barrault et al., 2019). NMT systems trained on back-translated data can generate more fluent translations (Sennrich et al., 2016a) thanks to the use of much larger data in the target language to better train the decoder, especially for low-resource conditions where only a small quantity of parallel training data is available. However, the impact of the noisiness of the synthetic source sentences gen- erated by NMT largely remains unclear and under- studied. Edunov et al. (2018) even showed that introducing synthetic noise in back-translations ac- tually improves translation quality and enables the use of a much larger quantity of back-translated data for further improvements in translation quality. More recently, Caswell et al. (2019) empirically demonstrated that adding a unique token at the beginning of each back-translation acts as a tag that helps the system during training to differenti- ate back-translated data from the original parallel training data and is as effective as introducing syn- thetic noise for improving translation quality. It is also much simpler since it requires only one edit- ing operation, adding the tag, and non-parametric. However, it is not fully understood why adding a tag has such a significant impact and to what extent it helps to distinguish back-translated data from the original parallel data. In this paper, we report on the impact of tag- ging back-translations in NMT, focusing on the following research questions (see Section 2 for our motivation). Q1. Do NMT systems trained on large back- translated data capture some of the charac- teristics of human-produced translations, i.e., translationese? Q2. Does a …

Content truncated to 500 words