MTee: Open Machine Translation Platform for Estonian Government

MTEE : Open Machine Translation Platform for Estonian Government Toms Bergmanis, M¯arcis Pinnis, Roberts Rozis, J¯anis ˇSlapin¸ˇs, Valters ˇSics, Berta Bern¯ane, Guntars Puˇzulis, Endijs Titomers Tilde, Latvia {name.surname}@tilde.lv Andre T¨attar, Taido Purason, Hele-Andra Kuulmets, Agnes Luhtaru, Liisa R¨atsep, Maali Tars, Annika Laumets-T¨attar, Mark Fishel University of Tartu, Estonia {name.surname}@ut.ee Abstract We present the MTEE project—a research initiative funded via an Estonian public procurement to develop machine transla- tion technology that is open-source and free of charge. The MTEE project de- livered an open-source platform serving state-of-the-art machine translation sys- tems supporting four domains for six lan- guage pairs translating from Estonian into English, German, and Russian and vice- versa. The platform also features gram- matical error correction and speech trans- lation for Estonian and allows for format- ted document translation and automatic domain detection. The software, data and training workflows for machine translation engines are all made publicly available for further use and research. 1 Project Background MTEE is an Estonian governmental project to de- velop high-quality machine translation (MT) plat- form that is open-source and free of charge. The project was motivated by the COVID-19 pan- demic. It was aimed to address the country’s need for fast and cheap translation of information to and from Estonian and the languages most relevant to Estonia’s society: English, German, and Russian. MTEE was funded by the Ministry of Education and Research via a public procurement through the Language Technology Competence Center at the Institute of the Estonian Language. The dura- tion of MTEE project was nine months, and it con- © 2022 The authors. This article is licensed under a Creative Commons 3.0 licence, no derivative works, attribution, CC- BY-ND. cluded in January 2022. It was fulfilled as a col- laboration between Tilde and the Institute of Com- puter Science of the University of Tartu. A demon- stration of the platform1 is made publicly available by hosting using the infrastructure of the High Per- formance Computing Center of the University of Tartu. 2 Data To train MT systems, we used parallel data from OPUS (Tiedemann, 2009), ELRC-SHARE (Piperidis et al., 2018) and EU Open Data Portal,2 as well as data donors and industry partners. In contrast, monolingual data were mainly obtained from the public web. To classify data as belong- ing to legal, military, crisis, or general domains, we used its source information. Furthermore, we used terminology provided by the Institute of the Estonian Language to automatically obtain addi- tional data for individual domains. The result- ing data sets ranged from 5 to 20 million parallel sentences for the general domain. However, data sets were much smaller for niche domains and lan- guage pairs, such as the German–Estonian crisis domain, where only a few dozen sentence pairs were identified. We observed a similar pattern for the monolingual data, for which data sizes ranged from 50 million sentences for the general domain to only 8 thousand sentences for the Russian mili- tary domain. We used random held-out subsets of training data for testing …

Content truncated to 500 words