Surprise Language Challenge: Developing a Neural Machine Translation System between Pashto and English in Two Months

Surprise Language Challenge: Developing a Neural Machine Translation System between Pashto and English in Two Months Alexandra Birch,1 Barry Haddow,1 Antonio Valerio Miceli Barone,1 Jindˇrich Helcl,1 Jonas Waldendorf,1 Felipe S´anchez-Mart´ınez,2 Mikel L. Forcada,2 Miquel Espl`a-Gomis,2 V´ıctor M. S´anchez-Cartagena,2 Juan Antonio P´erez-Ortiz,2 Wilker Aziz,3 Lina Murady,3 Sevi Sariisik,4 Peggy van der Kreeft,5 Kay MacQuarrie5 1University of Edinburgh, 2Universitat d’Alacant, 3Universiteit van Amsterdam, 4BBC , 5Deutsche Welle Abstract In the media industry, the focus of global reporting can shift overnight. There is a compelling need to be able to develop new machine translation systems in a short period of time, in or- der to more efficiently cover quickly developing stories. As part of the low-resource machine translation project GoURMET, we selected a surprise language for which a system had to be built and evaluated in two months (February and March 2021). The language selected was Pashto, an Indo-Iranian language spoken in Afghanistan, Pakistan and India. In this period we completed the full pipeline of development of a neural machine translation system: data crawl- ing, cleaning, aligning, creating test sets, developing and testing models, and delivering them to the user partners. In this paper we describe the rapid data creation process, and experiments with transfer learning and pretraining for Pashto-English. We find that starting from an existing large model pre-trained on 50 languages leads to far better BLEU scores than pretraining on one high-resource language pair with a smaller model. We also present human evaluation of our systems, which indicates that the resulting systems perform better than a freely available commercial system when translating from English into Pashto direction, and similarly when translating from Pashto into English. 1 Introduction The Horizon 2020 European-Union-funded project GoURMET1 (Global Under-Resourced MEdia Translation) aims to improve neural machine translation for under-resourced language pairs with a special emphasis on the news domain. The two media partners in the GoURMET project, the BBC in the UK and Deutsche Welle (DW) in Germany, publish news content in 40 and 30 different languages, respectively, and gather news in over 100 languages. In such a global information scenario, machine translation technologies become an important element in the everyday workflow of these media organisations. 1https://GoURMET-project.eu/ Proceedings of the 18th Biennial Machine Translation Summit Virtual USA, August 16 - 20, 2021, Volume 1: MT Research Track Page 92 Surprise language exercises (Oard et al., 2019) started in March 2003, when the US De- fense Advanced Research Projects Agency (DARPA) designated Cebuano, the second most widely spoken indigenous language in the Philippines, as the focus of an exercise. Teams were given only ten days to assemble language resources and to create whatever human language technology they could in that time. These events have been running annually ever since. The GoURMET project undertook its surprise language evaluation as an exercise to bring together the whole consortium to focus on a language pair of particular interest to the BBC and DW for a short period of time. Given the impact of the COVID-19 pandemic, a two- month …

Content truncated to 500 words