MATEO: MAchine Translation Evaluation Online

MATEO: MAchine Translation Evaluation Online Bram Vanroy, Arda Tezcan, Lieve Macken LT3, Language and Translation Technology Team Ghent University Belgium {firstname.lastname}@ugent.be Abstract We present MAchine Translation Evalua- tion Online (MATEO), a project that aims to facilitate machine translation (MT) eval- uation by means of an easy-to-use web interface that can evaluate given machine translations with a battery of automatic metrics. It caters to both experienced and novice users who are working with MT, such as MT system builders, teachers and students of (machine) translation, and re- searchers. 1 Introduction Due to the swift development of evaluation metrics for machine translation (MT) and the absence of up-to-date and user-friendly interfaces, this project aims to bridge the gap by joining together a diverse set of automatic, reference-based MT evaluation metrics, including both established and cutting- edge methods, into a single, easily accessible web interface. It is intended for researchers and prac- titioners in the Social Sciences and Humanities (SSH) and beyond, also including MT developers and researchers, translation scholars, and experts in the fields of digital humanities and (computa- tional) social sciences. Furthermore, the tool can serve as an instructional resource for educators and students because it emphasises the importance of evaluating language resources. It improves the dig- ital literacy of users: being able to easily evaluate machine-generated translations should make users aware not to blindly use MT systems but critically evaluate them for a task, topic, or domain at hand. © 2023 The authors. This article is licensed under a Creative Commons 4.0 licence, no derivative works, attribution, CC- BY-ND. MATEO’s web interface is open-source,1 GPLv3 licensed, and will be hosted at CLARIN.eu infras- tructure. This project was kick-started with a Sponsorship 2021 grant from the European Association of Ma- chine Translation. A substantial follow-up grant was acquired from the CLARIN.eu Bridging Gaps initiative. The secured funding accounts for half- time employment for one year at Ghent University for the first author of this paper, who is the devel- oper of this project. The project will end at the end of June 2023. 2 Related Platforms Similar platforms exist but they are either not maintained or do not provide all the functional- ity that we are interested in providing. In the past we made use of Asiya Online2 for teaching MT classes, which provided similar functionality as we are aiming for but unfortunately the service does not work anymore. It also does not support more recent metrics which we would like to include. Tilde MT also provides an interface to evaluate MT but it is limited to BLEU.3 MT-ComparEval (Kle- jch et al., 2015) is an open-source tool that is sim- ilar to our plans but it is rather dated when com- pared to the current, rapidly evolving landscape of MT evaluation metrics by only providing BLEU, precision, recall and F-scores.4 Finally, MutNMT provides an interface to train MT systems in an educational setting but its evaluation methods are limited to BLEU, TER and ChrF.5 1https://github.com/BramVanroy/mateo-demo 2https://asiya.cs.upc.edu/demo/asiya_ online.php 3https://www.letsmt.eu/Bleu.aspx 4https://ufal.mff.cuni.cz/tools/ mt-compareval …

Content truncated to 500 words