Measuring the Effects of Human and Machine Translation on Website Engagement

Measuring the Effects of Human and Machine Translation on Website Engagement Geza Kovacs geza@alumni.stanford.edu John DeNero john@lilt.com Lilt Inc, San Francisco, CA, USA Abstract With the internet growing increasingly multilingual, it is important to consider translating websites. However, professional translators are much more expensive than machines, and machine translation quality is continually increasing, so we must justify the cost of professional translation by measuring the effects of translation on website engagement, and how users interact with translations. This paper presents an in-the-wild study run on 2 websites fully translated into 15 and 11 languages respectively, where visitors with non-English preferred languages were randomized into being shown text translated by a professional translator, machine translated text, or untranslated English text. We find that both human and machine translations improve engagement, users rarely switch the page language manually, and that in-browser machine translation is often used when English is shown, particularly by users from countries with low English proficiency. We also release a dataset of interaction data collected during our studies, including 3,332,669 sessions from 190 countries across 2 websites. 1 Introduction The userbase of the internet is becoming increasingly linguistically diverse (Group, 2020). As a result, publishers increasingly need to translate websites to make content available to global audiences. Professional translators can be expensive, so localization decisions involve many tradeoffs. Should we show translations generated by professional translators, or is machine translation quality sufficiently high that showing machine translations will not negatively impact user engagement? If a human translation is not available in the user’s preferred language, is it preferable to show English or a machine translation? Does the website need to be translated at all, given that browsers have built-in machine translation functionality? To make informed decisions, we need to know the effects of each form of translation on website engagement. This paper quantifies the effects of translation on engagement metrics and how users interact with translations, by running large-scale (over 3 million sessions), in-the-wild A/B tests on the homepages of two different open-source software projects. Our contributions in this paper are a set of studies quantifying how users interact with website-provided translations and in- browser machine translations, interface recommendations on how to incorporate translations into websites based on these findings, and a public dataset of interactions and code that can be used to reproduce our results and conduct follow-up research. 2 Related Work Previously published work has not evaluated the effects of machine translation, human translation, and non-translation on in-the-wild website user engagement in a randomized, A/B test fashion. Proceedings of the 15th Biennial Conference of the Association for Machine Translation in the Americas Orlando, USA, September 12-16, 2022. Volume 1: Research Track 298 While machine translation quality has historically been negatively perceived (Läubli and Orrego- Carmona, 2017), some machine translation systems claim to have reached parity in translation quality with professional translators in certain settings (Hassan et al., 2018; Barrault et al., 2019; Popel et al., 2020). Other work has questioned these claims of human parity (Läubli et …

Content truncated to 500 words