Optimizing Transformer for Low-Resource Neural Machine Translation

Proceedings of the 28th International Conference on Computational Linguistics, pages 3429–3435 Barcelona, Spain (Online), December 8-13, 2020 3429 Optimizing Transformer for Low-Resource Neural Machine Translation Ali Araabi Informatics Institute University of Amsterdam a.araabi@uva.nl Christof Monz Informatics Institute University of Amsterdam c.monz@uva.nl Abstract Language pairs with limited amounts of parallel data, also known as low-resource languages, remain a challenge for neural machine translation. While the Transformer model has achieved significant improvements for many language pairs and has become the de facto mainstream ar- chitecture, its capability under low-resource conditions has not been fully investigated yet. Our experiments on different subsets of the IWSLT14 training data show that the effectiveness of Transformer under low-resource conditions is highly dependent on the hyper-parameter settings. Our experiments show that using an optimized Transformer for low-resource conditions improves the translation quality up to 7.3 BLEU points compared to using the Transformer default settings. 1 Introduction Despite the success of Neural Machine Translation (NMT) (Sutskever et al., 2014; Cho et al., 2014; Bahdanau et al., 2015), for the vast majority of language pairs for which only limited amounts of training data exist (a.k.a. low-resource languages), the performance of NMT systems is relatively poor (Koehn and Knowles, 2017; Gu et al., 2018a). Most approaches focus on exploiting additional data to address this problem (G¨ulc¸ehre et al., 2015; Sennrich et al., 2016; He et al., 2016; Fadaee et al., 2017). However, Sennrich and Zhang (2019) show that a well-optimized NMT system can perform relatively well under low-resource data conditions. Unfortunately, their results are confined to a recurrent NMT architecture (Sennrich et al., 2017), and it is not clear to what extent these findings also hold for the nowadays much more commonly used Transformer architecture (Vaswani et al., 2017). Like all NMT models, Transformer requires setting various hyper-parameters but researchers often stick to the default parameters, even when their data conditions differ substantially from the original data conditions used to determine those default values (Gu et al., 2018b; Aharoni et al., 2019). In this paper, we explore to what extent hyper-parameter optimization, which has been applied suc- cessfully to recurrent NMT models for low-resource translation, is also beneficial for the Transformer model. We show that with the appropriate settings, including the number of BPE merge operations, attention heads, and layers up to the degree of dropout and label smoothing, translation performance can be increased substantially, even for data sets with as little as 5k sentence pairs. Our experiments on different corpus sizes, ranging from 5k to 165k sentence pairs, show the importance of choosing the optimal settings with respect to data size. 2 Hyper-Parameter Exploration In this section, we first discuss the importance of choosing an appropriate degree of subword segmenta- tion before we describe the other optimal hyper-parameter settings. Vocabulary representation. In order to improve the translation of rare words, word segmentation ap- proaches such as Byte-Pair-Encoding (BPE) (Sennrich et al., 2016) have become standard practice in NMT. This is especially true for language pairs with small amounts of …

Content truncated to 500 words