What is the impact of raw MT on Japanese users of Word: preliminary results of a usability study using eye-tracking

What is the impact of raw MT on Japanese users of Word: preliminary results of a usability study using eye-tracking Ana Guerberof ADAPT/SALIS Dublin City University ana.guerberof@dcu.ie Joss Moorkens ADAPT/SALIS Dublin City University joss.moorkens@dcu.ie Sharon O’Brien ADAPT/SALIS Dublin City University sharon.obrien@dcu.ie Abstract This paper presents preliminary results of a study of Japanese native speakers working with the Microsoft Word application in two modalities: the released Japanese version and a machine translated (MT) version (the raw MT strings incorporated into the MS Word interface). To explore the effect of translation modality on task completion, time and satisfaction, an experiment using an eye- tracker was set up with a group of 42 users: 22 native Japanese and 20 native English speakers. The results suggest that Japanese-native speakers have higher completion scores and are more efficient when working with the released versions of the product than with the MT version, but these differences are not significant. Their self-reported satisfaction, however, is significantly higher when working with the released product as opposed to the raw MT version. 1 Introduction In the commercial arena, the software and localization industries face long-term business challenges. There is an increase in the volume of software to localize, and this software needs to run on several platforms. Moreover, the software is delivered to the user in a rapid cycle, with daily, weekly, and quarterly updates and releases. In parallel, there are continuous advances in machine translation (MT) technology with the full implementation of statistical engines and rapid advances in neural MT solutions. Therefore, it is only logical to marry the use of new technology with localization of software products with the aid © 2019 The authors. This article is licensed under a Creative Commons 4.0 licence, no derivative works, attribution, CCBY-ND. of automation where possible, as long as this can be achieved without hindering the user experience and, hence, the commercial viability of that product. Large software corporations have, in fact, implemented MT and post-editing (PE) cycles as part of their localization processes for some time now. However, as MT technology advances, several questions come to mind: is it possible to apply raw (i.e. unedited) MT to certain components of the user interface without hindering the user experience? Where raw MT is employed, how does linguistic quality impact the user experience? Preliminary results are presented here from a usability experiment involving Japanese and English native speakers using an eye-tracker. The Japanese participants were presented with two Microsoft Word applications: one was the Japanese-released version (referred as HT hereafter), and the other one was a version translated from English into Japanese with MT specifically for this experiment (referred as MT hereafter). English speakers were presented with one Word application (to act as the control group). The different versions (HT, MT and English) are referred as scenarios. Both groups had to perform the same six tasks. 2 Related work MT and PE have been implemented in some large organizations since the 1980s (the European Commission and the Pan American Health Organization, …

Content truncated to 500 words