Benchmarking Neural and Statistical Machine Translation on Low-Resource African Languages

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 2667–2675 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 2667 Benchmarking Neural and Statistical Machine Translation on Low-Resource African Languages Kevin Duh, Paul McNamee, Matt Post, Brian Thompson Johns Hopkins University Baltimore, MD, USA {kevinduh, post}@cs.jhu.edu, {mcnamee,brian.thompson}@jhu.edu Abstract Research in machine translation (MT) is developing at a rapid pace. However, most work in the community has focused on languages where large amounts of digital resources are available. In this study, we benchmark state of the art statistical and neural machine translation systems on two African languages which do not have large amounts of resources: Somali and Swahili. These languages are of social importance and serve as test-beds for developing technologies that perform reasonably well despite the low-resource constraint. Our findings suggest that statistical machine translation (SMT) and neural machine translation (NMT) can perform similarly in low-resource scenarios, but neural systems require more careful tuning to match performance. We also investigate how to exploit additional data, such as bilingual text harvested from the web, or user dictionaries; we find that NMT can significantly improve in performance with the use of these additional data. Finally, we survey the landscape of machine translation resources for the languages of Africa and provide some suggestions for promising future research directions. Keywords: machine translation, low-resource languages, evaluation 1. Introduction Parallel text is an essential ingredient for building Statisti- cal Machine Translation (SMT) and Neural Machine Trans- lation (NMT) systems. By definition, parallel text is a kind of corpus consisting of pairs of sentences, where one is written in the source language (e.g., Somali) and the other is its translation in the target language (e.g., English). This is an expensive resource to manually generate, requiring translators that are proficient in both languages. Commercial SMT and NMT systems are often trained on millions to tens of millions of sentence pairs, if not more (Wu et al., 2016). It is unclear how these systems perform when the training data contains significantly fewer sentence pairs. For many languages in the world, and in particu- lar for languages in the African continent, at present we cannot reasonably expect such a large amount of training data. While there is no established convention, we might consider systems that are trained on less than 100 thousand sentence pairs to be low-resource. Previous work (Koehn and Knowles, 2017; Sennrich and Zhang, 2019) has established the idea that there is a cross- over point between NMT and SMT performance depending on the amount of training data. See Figure 1. The intuition is that NMT is data-hungry, so may perform worse than SMT in low-resource settings, but begins to excel when there is sufficient training data. With recent advances in NMT, the cross-over point has gradually decreased. Never- theless, in general it is difficult to predict a priori whether we are on the left or right side of the cross-over point until we actually build the systems. In this …

Content truncated to 500 words