The Effect of Domain and Diacritics in Yoruba–English Neural Machine Translation

The Effect of Domain and Diacritics in Yor`ub´a–English Neural Machine Translation David Ifeoluwa Adelani∗ didelani@lsv.uni-saarland.de Spoken Language Systems Group (LSV), Saarland University, Germany & Masakhane NLP Dana Ruiter∗ druiter@lsv.uni-saarland.de Spoken Language Systems Group (LSV), Saarland University, Germany Jesujoba O. Alabi∗ jalabi@mpi-inf.mpg.de Max Planck Institute for Informatics, Saarbr¨ucken, Germany & Masakhane NLP Damilola Adebonojo iyayorubagidi@gmail.com Alamoja Yoruba & Masakhane NLP Adesina Ayeni info@yobamoodua.org Yobamoodua Cultural Heritage (YMCH) Mofe Adeyemi mofetoluwa@outlook.com Defence Space Administration, Abuja, Nigeria & Masakhane NLP Ayodele Awokoya ayodeleawokoya@gmail.com University of Ibadan, Nigeria & Masakhane NLP Cristina Espa˜na-Bonet cristinae@dfki.de DFKI GmbH, Saarland Informatics Campus, Saarbr¨ucken, Germany Abstract Massively multilingual machine translation (MT) has shown impressive capabilities, including zero and few-shot translation between low-resource language pairs. However, these models are often evaluated on high-resource languages with the assumption that they generalize to low-resource ones. The difficulty of evaluating MT models on low-resource pairs is often due to lack of standardized evaluation datasets. In this paper, we present MENYO-20k, the first multi-domain parallel corpus with a special focus on clean orthography for Yor`ub´a–English with standardized train-test splits for benchmarking. We provide several neural MT bench- marks and compare them to the performance of popular pre-trained (massively multilingual) MT models both for the heterogeneous test set and its subdomains. Since these pre-trained models use huge amounts of data with uncertain quality, we also analyze the effect of dia- critics, a major characteristic of Yor`ub´a, in the training data. We investigate how and when this training condition affects the final quality and intelligibility of a translation. Our models outperform massively multilingual models such as Google (+8.7 BLEU) and Facebook M2M (+9.1 BLEU) when translating to Yor`ub´a, setting a high quality benchmark for future research. ∗* Equal contribution to the work Proceedings of the 18th Biennial Machine Translation Summit Virtual USA, August 16 - 20, 2021, Volume 1: MT Research Track Page 61 1 Introduction Neural machine translation (NMT) achieves high quality performance when large amounts of parallel sentences are available (Barrault et al., 2020). Large and freely-available parallel cor- pora do exist for a small number of high-resource pairs and domains. However, for low-resource languages such as Yor`ub´a (yo), one can only find few thousands of parallel sentences online1. In the best-case scenario, i.e. some amount of parallel data exists, one can use the Bible — the Bible is the most available resource for low-resource languages (Resnik et al., 1999)— and JW300 (Agi´c and Vuli´c, 2019). Notice that both corpora belong to the religious domain and they do not generalize well to popular domains such as news and daily conversations. In this paper, we address this problem for the Yor`ub´a–English (yo–en) language pair by creating a multi-domain parallel dataset, MENYO-20k, which we make publicly available2 with CC BY-NC 4.0 licence. It is a heterogeneous dataset that comprises texts obtained from news articles, TED talks, movie and radio transcripts, science and technology texts, and other short articles curated from the web and translated by professional translators. Based on the resulting train-development-test split, we provide …

Content truncated to 500 words