How well do real-time machine translation apps perform in practice? Insights from a literature review

How well do real-time machine translation apps perform in practice? Insights from a literature review Mark Pluymaekers Zuyd University of Applied Sciences Brusselseweg 150, 6217 HB Maastricht (NL) mark.pluymaekers@zuyd.nl Abstract Although more and more professionals are using real-time machine translation during dialogues with interlocutors who speak a different language, the performance of real-time MT apps has received only lim- ited attention in the academic literature. This study summarizes the findings of prior studies (N = 34) reporting an evalu- ation of one or more real-time MT apps in a professional setting. Our findings show that real-time MT apps are often tested in realistic circumstances and that users are more frequently employed as judges of performance than professional translators. Furthermore, most studies report overall positive results with regard to perfor- mance, particularly when apps are tested in real-life situations. 1 Introduction In 1997, Mark Seligman wrote that “the Internet offers a tremendous opportunity for experiments with real-time machine translation (MT) of dialogues” (Seligman, 1997). In December of the same year, SYSTRAN and AltaVista launched “the first widely available, real-time, high-speed and free translation service on the Internet” (Yang & Lange, 1998). Now, 25 years later, the Google Translate app has been downloaded more than 1 billion times from the Google App Store (Pitman, 2021). Since 2011, the app offers a conversation mode, which enables users to have utterances within a dialogue translated in real-time so that their conversation partners can understand them. Other apps such as iTranslate, TripLingo and © 2022 The author. This article is licensed under a Creative Commons 3.0 licence, no derivative works, attribution, CC-BY-ND. Microsoft Translator can also be used to support synchronous dialogue between interlocutors who do not speak the same language (Tao, 2022). To the best of our knowledge, there are no publicly available data on the frequency with which MT apps are used for real-time translation and the contexts in which this occurs. However, given the popularity of these apps, it can be expected that a large number of synchronous dialogues are translated every day, and that this happens not only in informal situations, but also in professional contexts. This raises the question of how well real-time MT apps perform in these kinds of situations. Traditionally, the academic literature has paid more attention to the quality of written translations that have been produced using MT than to the output of real-time MT apps. This study aims to boost research into the performance of real-time MT apps by summarizing the findings of earlier studies in which the performance of such apps was evaluated in a professional context. 2 MT quality assessment The quality of MT output has been a hotly debated topic for decades, and a wide variety of methods for its assessment have been proposed (cf. Castilho et al., 2018). When classifying these methods, authors commonly distinguish between automated metrics and human metrics (e.g., Rivera-Trigueros, 2021; Chatzikoumi, 2020). Automated metrics include Word Error Rates (WERs), precision, recall, and BLEU scores, all of which are …

Content truncated to 500 words