Self-Supervised Neural Machine Translation

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1828–1834 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 1828 Self-Supervised Neural Machine Translation Dana Ruiter Cristina Espa˜na-Bonet Josef van Genabith Saarland University Saarland University Saarland University DFKI GmbH druiter@lsv.uni-saarland.deDFKI GmbH {cristinae,Josef.Van Genabith}@dfki.de Abstract We present a simple new method where an emergent NMT system is used for simultane- ously selecting training data and learning in- ternal NMT representations. This is done in a self-supervised way without parallel data, in such a way that both tasks enhance each other during training. The method is lan- guage independent, introduces no additional hyper-parameters, and achieves BLEU scores of 29.21 (en2fr) and 27.36 (fr2en) on new- stest2014 using English and French Wikipedia data for training. 1 Introduction Neural machine translation (NMT) has brought major improvements in translation quality (Cho et al., 2014; Bahdanau et al., 2014; Vaswani et al., 2017). Until recently, these relied on the avail- ability of high-quality parallel corpora. As such corpora exist only for a few high-resource lan- guage combinations, overcoming this constraint by either extracting parallel data from non-parallel sources or developing unsupervised techniques in NMT is crucial to cover all languages. Obtaining comparable corpora is becoming eas- ier (Paramita et al., 2019) and extracting par- allel sentences from them a wide research field. Most of the methods estimate similarities between fragments to select pairs. Here we focus on similarities estimated from NMT representations. The strength of NMT embeddings as semantic representations was first shown qualitatively in Sutskever et al. (2014); Ha et al. (2016) and John- son et al. (2017), and used for estimating semantic similarities at sentence level in Espa˜na-Bonet and Barr´on-Cede˜no (2017) for example. In a system- atic study, Espa˜na-Bonet et al. (2017) show that cosine similarities between context vectors dis- criminate between parallel and non-parallel sen- tences already in the first stages of training. Other approaches perform max-pooling over encoder outputs (Schwenk, 2018; Artetxe and Schwenk, 2018) or calculate the mean of word embeddings (Bouamor and Sajjad, 2018) to extract pairs. On the other hand, unsupervised NMT is now achieving impressive results using large amounts of monolingual data and small parallel lexicons (Lample et al., 2018a; Artetxe et al., 2018b; Yang et al., 2018). These systems rely on very strong language models and back-translation, and build complex architectures that combine denoising au- toencoders, back-translation steps and shared en- coders among languages. The most successful architectures also use SMT phrase tables, stand- alone or in combination with NMT (Lample et al., 2018b; Artetxe et al., 2018a). In our approach, we propose a new and sim- pler method without a priori parallel corpora. Our premise is that NMT systems —either sequence to sequence models with RNNs, transformers, or any architecture based on encoder–decoder models— already learn strong enough representations of words and sentences to judge on-line if an input sentence pair is useful or not. Our approach re- sembles self-supervised learning (Raina et al., 2007; Bengio et al., …

Content truncated to 500 words