Evaluating the Impact of Light Post-Editing on Usability

Evaluating the Impact of Light Post-Editing on Usability Sheila Castilho and Sharon O’Brien Adapt Centre/SALIS. Dublin City University Ireland E-mail: castils3@mail.dcu.ie, sharon.obrien2@dcu.ie Abstract This paper discusses a methodology to measure the usability of machine translated content by end users, comparing lightly post-edited content with raw output and with the usability of source language content. The content selected consists of Online Help articles from a software company for a spreadsheet application, translated from English into German. Three groups of five users each used either the source text - the English version (EN) -, the raw MT version (DE_MT), or the light PE version (DE_PE), and were asked to carry out six tasks. Usability was measured using an eye tracker and cognitive, temporal and pragmatic measures of usability. Satisfaction was measured via a post-task questionnaire presented after the participants had completed the tasks. Keywords: post-editing, usability, eye tracking 1. Introduction Recent advances in machine translation (MT) have enabled post-editing (PE) to become a more common practice in the translation industry, which has led to much research in the area (De Almeida and O’Brien, 2010; Depraetere, 2010; Plitt and Masselot, 2010; Sousa et al., 2011; Specia, 2011; Koponen, 2012; O’Brien et al., 2013; Guerberof, 2014; Moorkens et al., 2015). However, we know little about how end users engage with raw machine-translated text or post-edited text, or how usable such texts are, in particular if users have to follow instructions and subsequently act on them. This paper reports on a methodology to measure usability of machine translation output. The main objectives of this study are: i) to investigate the extent to which light human post-editing of machine translation impacts on the usability of instructional, online help content and, ii) to compare this with usability levels of the source text. The paper is structured as follows: Section 2 discusses related research, Section 3 and Section 4 describe the content used and the participants of the experiment respectively, Section 5 discusses the methods deployed to measure usability, Section 6 provides the preliminarily results, while Section 7 presents conclusions and plans for future work. 2. Related Work Existing work measuring the usability of machine translated content is still somewhat limited. Tomita et al. (1993) compare different MT systems by using reading comprehension tests from texts extracted from an English proficiency exam and translated into Japanese. They show that reading comprehension is a valid evaluation methodology for MT. Fuji et al. (2001) examine the “usefulness” of machine translated text from two commercial MT systems compared to the original English version. The experiment consists of participants reading the texts and answering comprehension questions. Afterwards, participants evaluate the MT outputs on a 5-point scale using comprehensibility and awkwardness as concepts. Results suggest that the MT output reduces the time to answer questions for the lower score group. The authors claim their evaluation approach delivers statistically significant results easily understood by the general public. Jones et al. (2005) present a usability test where participants answer questions from a machine translated version …

Content truncated to 500 words