In Neural Machine Translation, What Does Transfer Learning Transfer?

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7701–7710 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 7701 In Neural Machine Translation, What Does Transfer Learning Transfer? Alham Fikri Aji1 Nikolay Bogoychev1 Kenneth Heafield1 Rico Sennrich2,1 1School of Informatics, University of Edinburgh 2Department of Computational Linguistics, University of Zurich {a.fikri, n.bogoych, kenneth.heafield, rico.sennrich}@ed.ac.uk Abstract Transfer learning improves quality for low- resource machine translation, but it is unclear what exactly it transfers. We perform several ablation studies that limit information trans- fer, then measure the quality impact across three language pairs to gain a black-box un- derstanding of transfer learning. Word embed- dings play an important role in transfer learn- ing, particularly if they are properly aligned. Although transfer learning can be performed without embeddings, results are sub-optimal. In contrast, transferring only the embeddings but nothing else yields catastrophic results. We then investigate diagonal alignments with auto-encoders over real languages and ran- domly generated sequences, finding even ran- domly generated sequences as parents yield noticeable but smaller gains. Finally, transfer learning can eliminate the need for a warm- up phase when training transformer models in high resource language pairs. 1 Introduction Transfer learning is a common method for low- resource neural machine translation (NMT) (Zoph et al., 2016; Dabre et al., 2017; Qi et al., 2018; Nguyen and Chiang, 2017; Gu et al., 2018b). How- ever, it is unclear what settings make transfer learn- ing successful and what knowledge is being trans- ferred. Understanding why transfer learning is success- ful can improve best practices while also opening the door to investigating ways to gain similar ben- efits without requiring parent models. In this pa- per, we perform several ablation studies on transfer learning in order to understand what information is being transferred. We apply a black box methodology by measur- ing the quality of end-to-end translation systems. Typically, our experiments have a baseline that was trained from scratch, an off-the-shelf transfer learn- ing baseline and simplified versions of the transfer learning scheme. If a simplified version recovers some of the quality gains of full transfer learning, it suggests that the simplified version has captured some of the information being transferred. Since information may be transferred redundantly, our claims are limited to sufficiency rather than exclu- sivity. Transferring word embeddings is not straight- forward since languages have different vocabular- ies. Zoph et al. (2016) claimed that vocabulary alignment is not necessary, while Nguyen and Chi- ang (2017) and Kocmi and Bojar (2018) suggest a joint vocabulary. We find that the vocabulary has to be aligned before transferring the embedding to achieve a substantial improvement. Transfer learning without the embedding or with vocabulary mismatches is still possible, but with lower quality. Conversely, transferring only the word embeddings can be worse than transferring nothing at all. A rudimentary model of machine translation con- sists of alignment and token mapping. We hypoth- esize that these capabilities are transferred across languages. To test this, we experiment with trans- ferring …

Content truncated to 500 words