Does post-editing increase usability? A study with Brazilian Portuguese as target language

Does post-editing increase usability? A study with Brazilian Portu- guese as Target Language Sheila Castilho CNGL/SALIS Dublin City University Ireland castils3@mail.dcu.ie Sharon O’Brien CNGL/SALIS Dublin City University Ireland sharon.obrien@dcu.ie Fabio Alves Federal University of Minas Gerais (UFMG) Brazil fabio-alves@ufmg.br Morgan O’Brien McAfee Mahon, Cork Ireland morgan_o'brien@mcafee.com Abstract It is often assumed that raw MT output requires post-editing if it is to be used for more than gisting purposes. However, we know little about how end users engage with raw machine translated text or post- edited text, or how usable this text is, in particular if users have to follow instruc- tions and act on them. The research pro- ject described here measures the usability of raw machine translated text for Brazil- ian Portuguese as a target language and compares that with a post-edited version of the text. Two groups of 9 users each used either the raw MT or the post-edited version and carried out tasks using a PC- based security product. Usability was measured using an eye tracker and cogni- tive, temporal and pragmatic measures of usability, and satisfaction was measured using a post-task questionnaire. Results indicate that post-editing significantly in- creases the usability of machine translat- ed text. 1 Introduction This paper discusses the measurement of usabil- ity for raw machine translated output and post- edited output for instructional text relating to a commercial PC security product machine trans- lated from English into Brazilian Portuguese. © 2014 European Association for Machine Translation. Authentic English source text relating to the software product (anonymised for confidentiality reasons) was identified and machine translated into Brazilian Portuguese using the freely availa- ble MT engine, Microsoft Bing. Eighteen users were recruited to read the in- structions and carry out tasks by creating files and folders, changing settings within the product etc. The participants were divided equally into two groups; one group used the raw machine translated instructions and the other used the post-edited instructions. The usability of both sets of instructions was investigated using screen recording, eye tracking and a post-task question- naire. The main objective of this project was to investigate the extent to which human post- editing of machine translation impacted on the usability of instructional content.1 The paper is structured as follows: Section 2 discussed related research, Section 3 explains the methods used, Section 4 provides results and Section 5 the conclusions. 2 Related Work The task and process of post-editing has received significant attention in the past few years (e.g. Guerberof (2014), De Almeida and O‟Brien (2010), Depraetere (2010), Plitt and Masselot (2010), Sousa et al. (2011), Koponen (2012), O‟Brien et al. (2012), O‟Brien et al. (2013), Spe- cia (2011)). While MT technology has made sig- 1 This research is supported by the International Stra- tegic Cooperation Award through Science Foundation Ireland and Dublin City University. 183 nificant strides in the last decade, it is accepted that post-editing is needed in cases where the content is required for more than gisting purpos- es. Empirical research has demonstrated that post-editing …

Content truncated to 500 words