Adapting Machine Translation Education to the Neural Era: A Case Study of MT Quality Assessment

Adapting Machine Translation Education to the Neural Era: A Case Study of MT Quality Assessment Lieve Macken, Bram Vanroy and Arda Tezcan LT3, Language and Translation Technology Team Ghent University Belgium {firstname.lastname}@ugent.be Abstract The use of automatic evaluation metrics to assess Machine Translation (MT) quality is well established in the translation indus- try. Whereas it is relatively easy to cover the word- and character-based metrics in an MT course, it is less obvious to integrate the newer neural metrics. In this paper we discuss how we introduced the topic of MT quality assessment in a course for transla- tion students. We selected three English source texts, each having a different diffi- culty level and style, and let the students translate the texts into their L1 and reflect upon translation difficulty. Afterwards, the students were asked to assess MT quality for the same texts using different methods and to critically reflect upon obtained re- sults. The students had access to the MA- TEO web interface, which contains word- and character-based metrics as well as neu- ral metrics. The students used two differ- ent reference translations: their own trans- lations and professional translations of the three texts. We not only synthesise the comments of the students, but also present the results of some cross-lingual analyses on nine different language pairs. 1 Introduction Machine translation (MT) is increasingly be- ing used in professional translation workflows. “MT literacy and awareness of MT’s possibil- ities and limitations” forms therefore, accord- ing to the EMT competence framework (EMT, © 2023 The authors. This article is licensed under a Creative Commons 4.0 licence, no derivative works, attribution, CC- BY-ND. 2022), an integral part of professional transla- tion competences. At Ghent University, we offer a 5-credit course Machine Translation and Post- editing, which is part of the one-year postgraduate programme Computer-Assisted Language Media- tion1 and the two-year European Master in Tech- nology for Translation and Interpreting2. The MT part of the course aims to provide a comprehen- sive overview and covers topics such as the main linguistic challenges for MT, different approaches to MT (rule-based, statistical and neural MT) and MT evaluation. The students also acquire hands- on experience in building and evaluating their own MT systems using MutNMT3. The use of automatic evaluation metrics to as- sess Machine Translation (MT) quality is well es- tablished in the translation industry. The more traditional word- and character-based metrics are relatively easy to run and it is therefore easy to incorporate them in a university course. But, much more technical knowledge is required to get the neural metrics, which are based on large pre- trained language models, up and running. Despite their better performance, they are therefore less popular in translation courses. In this paper, we discuss how we introduced the topic of MT qual- ity assessment in a course for translation students of varying language backgrounds. The students got one dedicated lecture on the subject of MT evaluation, which covers both manual and auto- matic evaluation methods and had …

Content truncated to 500 words