Transformers for Low-Resource Languages: Is Féidir Linn!

Transformers for Low-Resource Languages: Is F´eidir Linn! S´eamus Lankford seamus.lankford@adaptcentre.ie ADAPT Centre, Department of Computing, Dublin City University, Dublin, Ireland. Haithem Afli haithem.afli@adaptcentre.ie ADAPT Centre, Department of Computer Science, Munster Technological University, Ireland. Andy Way andy.way@adaptcentre.ie ADAPT Centre, School of Computing, Dublin City University, Dublin, Ireland. Abstract The Transformer model is the state-of-the-art in Machine Translation. However, in general, neural translation models often under perform on language pairs with insufficient training data. As a consequence, relatively few experiments have been carried out using this architecture on low-resource language pairs. In this study, hyperparameter optimization of Transformer models in translating the low-resource English-Irish language pair is evaluated. We demon- strate that choosing appropriate parameters leads to considerable performance improvements. Most importantly, the correct choice of subword model is shown to be the biggest driver of translation performance. SentencePiece models using both unigram and BPE approaches were appraised. Variations on model architectures included modifying the number of layers, testing various regularisation techniques and evaluating the optimal number of heads for attention. A generic 55k DGT corpus and an in-domain 88k public admin corpus were used for evalua- tion. A Transformer optimized model demonstrated a BLEU score improvement of 7.8 points when compared with a baseline RNN model. Improvements were observed across a range of metrics, including TER, indicating a substantially reduced post editing effort for Transformer optimized models with 16k BPE subword models. Bench-marked against Google Translate, our translation engines demonstrated significant improvements. The question of whether or not Transformers can be used effectively in a low-resource setting of English-Irish translation has been addressed. Is f´eidir linn - yes we can. 1 Introduction The advent of Neural Machine Translation (NMT) has heralded an era of high-quality trans- lations. However, these improvements have not been manifested in the translation of all lan- guages. Large datasets are a prerequisite for high quality NMT. This works well in the context of well-resourced languages where there is an abundance of data. In the context of low-resource languages which suffer from a sparsity of data, alternative approaches must be adopted. An important part of this research involves developing applications and models to address the challenges of low-resource language technology. Such technology incorporates methods to address the data scarcity affecting deep learning for digital engagement of low-resource lan- guages. It has been shown that an out-of-the-box NMT system, trained on English-Irish data, achieves a lower translation quality compared with using a tailored SMT system (Dowling et Proceedings of the 18th Biennial Machine Translation Summit Virtual USA, August 16 - 20, 2021, Volume 1: MT Research Track Page 48 al, 2018). It is in this context that further research is required in the development of NMT for low-resource languages and the Irish language in particular. Most research on choosing subword models has focused on high resource languages (Ding et al., 2019; Gowda and May, 2020). In the context of developing models for English to Irish translation, there are no clear recommendations on the choice of subword model types. One …

Content truncated to 500 words