A Systematic Study Reveals Unexpected Interactions in Pre-Trained Neural Machine Translation

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 1437–1443 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 1437 A Systematic Study Reveals Unexpected Interactions in Pre-Trained Neural Machine Translation Ashleigh Richardson, Janet Wiles The University of Queensland Queensland, Australia {a.richardson, j.wiles}@uq.edu.au Abstract A significant challenge in developing translation systems for the world’s ∼7,000 languages is that very few have sufficient data for state-of-the-art techniques. Transfer learning is a promising direction for low-resource neural machine translation (NMT), but introduces many new variables which are often selected through ablation studies, costly trial-and-error, or niche expertise. When pre-training an NMT system for low-resource translation, the pre-training task is often chosen based on data abundance and similarity to the main task. Factors such as dataset sizes and similarity have typically been analysed independently in previous studies, due to the computational cost associated with systematic studies. However, these factors are not independent. We conducted a three-factor experiment to examine how language similarity, pre-training dataset size and main dataset size interacted in their effect on performance in pre-trained transformer-based low-resource NMT. We replicated the common finding that more data was beneficial in bilingual systems, but also found a statistically significant interaction between the three factors, which reduced the effectiveness of large pre-training datasets for some main task dataset sizes (p-value < 0.0018). The surprising trends identified in these interactions indicate that systematic studies of interactions may be a promising long-term direction for guiding research in low-resource neural methods. Keywords: Machine Translation, Evaluation Methodologies, Less-Resourced Languages, Transfer Learning, Pre-training 1. Introduction State-of-the-art techniques for neural machine trans- lation (NMT), such as transformers (Vaswani et al., 2017), have raised societal expectations of language technologies. However, the significant data require- ments for deep learning make it difficult to meet these expectations for many low-resource languages. This difference in access to technology exacerbates exist- ing systemic inequality. The nature of low-resource languages makes it difficult or impossible to collect enough data for blackbox application of these tech- niques; for instance, many languages have very few (or no) remaining speakers. In some cases, the hardware required to train or pre-train on hundreds of millions of sentence pairs is not available to the communities who require this technology. Nonetheless, the development of machine learning systems for these languages is in- teresting for reasons of cultural promotion, accessibil- ity, and as a means for testing the limits of technology. Transfer learning can be used to boost performance on a low-resource task, by leveraging information from one or more high-resource tasks. However, transfer learning introduces many new variables that are not trivial to decide. For instance, consider the simplest case of pre-training on an auxiliary language pair be- fore fine-tuning on the intended (main) language pair. In choosing an auxiliary task, we should consider task similarity, and amount of available data for both main and auxiliary tasks. However, the amount of computa- tionally expensive trial-and-error involved in choosing an appropriate auxiliary task is a …

Content truncated to 500 words