Reading Comprehension of Machine Translation Output: What Makes for a Better Read?

Reading Comprehension of Machine Translation Output: What Makes for a Better Read? Sheila Castilho ADAPT Centre Dublin City University sheila.castilho@adaptcentre.ie Ana Guerberof Arenas ADAPT Centre/SALIS Dublin City University ana.guerberof@adaptcentre.ie Abstract This paper reports on a pilot experiment that compares two different machine trans- lation (MT) paradigms in reading com- prehension tests. To explore a suitable methodology, we set up a pilot experi- ment with a group of six users (with En- glish, Spanish and Simplified Chinese lan- guages) using an English Language Test- ing System (IELTS), and an eye-tracker. The users were asked to read three texts in their native language: either the original English text (for the English speakers) or the machine-translated text (for the Span- ish and Simplified Chinese speakers). The original texts were machine-translated via two MT systems: neural (NMT) and sta- tistical (SMT). The users were also asked to rank satisfaction statements on a 3-point scale after reading each text and answering the respective comprehension questions. After all tasks were completed, a post-task retrospective interview took place to gather qualitative data. The findings suggest that the users from the target languages com- pleted more tasks in less time with a higher level of satisfaction when using transla- tions from the NMT system. 1 Introduction Recently, there has been an increase in Neural Ma- chine Translation (NMT) research as contempo- rary hardware supports much more powerful com- putation during the creation process. Research c⃝ 2018 The authors. This article is licensed under a Creative Commons 3.0 licence, no derivative works, attribution, CC- BY-ND. on the translation quality of NMT engines show that, in general, when compared against Statistical Machine Translation (SMT) engines, the output quality of NMT systems is higher when measured using automatic metrics (Bahdanau et al., 2014; Jean et al., 2015; Bojar et al., 2016; Koehn and Knowles, 2017). However, results are not as pos- itive when human evaluators compare these out- puts (Bentivogli et al., 2016; Castilho et al., 2017a; Castilho et al., 2017b). Human evaluation of MT output, although not always implemented in quality evaluation, has been increasingly endorsed by researchers who ac- knowledge the need for human assessments. Some of the most commonly-used manual metrics are fluency and adequacy, error analysis, translation ranking, as well as post-editing effort. Despite the considerable focus on MT quality evaluation, the impact of MT on the end user has been under- researched. Measuring the usability of MT out- put allows for identification of the impact that the translation might have on the end user (Castilho et al., 2014). With the intention of exploring the cognitive effort required to read texts originating from SMT and NMT engines by the end users of those texts, we set-up a pilot experiment that aims to measure the reading comprehension of Spanish and Simplified Chinese users of texts produced by both paradigms using an eye-tracker (using the En- glish users’ data as a baseline). The remainder of this paper is organized as fol- lows: in Section 2, we survey the existing …

Content truncated to 500 words