Post-Editing Evaluations: Trade-offs between Novice and Professional Participants

Post-Editing Evaluations: Trade-offs between Novice and Profes- sional Participants Joss Moorkens ADAPT Centre/School of Computing Dublin City University Ireland joss.moorkens@dcu.ie Sharon O’Brien ADAPT Centre/SALIS/CTTS Dublin City University Ireland sharon.obrien@dcu.ie Abstract The increasing use of post-editing in localisation workflows has led to a great deal of research and development in the area, much of it requiring user evaluation. This paper compares some results from a post-editing user inter- face study carried out using novice and expert translator groups. By comparing rates of productivity, edit distance, en- gagement with the research, and quali- tative findings regarding each group‟s attitude to post-editing, we find that there are trade-offs to be considered when selecting participants for evalua- tion tasks. Novices may generally be more positive and enthusiastic and will engage considerably with the research while professionals will be more effi- cient, but their routines and attitudes may prevent full engagement with re- search objectives. 1 Introduction The use of machine translation (MT) in com- mercial translation and localisation workflows has grown exponentially in recent years. Rela- tively recent breakthroughs in the quality of statistical machine translation (SMT) output has led to the use of MT for assimilation (gist- ing) and MT for dissemination (post-edited © 2015 The authors. This article is licensed under a Creative Commons 3.0 licence, no derivative works, attribution, CC-BY-ND. This research is supported by the Science Foundation Ireland (Grant 12/CE/I2267) as part of the CNGL (www.cngl.ie) at Dublin City Univer- sity and by the FALCON Project (falcon-project.eu), funded by the European Commission through the Sev- enth Framework Programme (FP7) Grant Agreement No. 610879. MT). The growth in the amount of content to be translated and a push for cost-cutting from translation clients has meant that post-editing of MT has grown in popularity – a survey of almost 1000 language service providers (LSPs) in 2013 found that over 44% offer a post-editing (PE) service to customers (De- Palma et al., 2013). This has led to a requirement for user test- ing, as industry and researchers attempt to learn how translators work with MT, through the task of post-editing, and most usually with- in a translation memory tool (Moorkens and O‟Brien, 2013). User dissatisfaction with post- editing has been widely reported (Krings, 2001; O‟Brien and Moorkens, 2014) and trans- lators tend to associate translation automation negatively with “regimentation, dependence, exploitation or impotence” (Cronin, 2013). Any new features intended to make the task more palatable to translators will naturally need to be tested for effectiveness. Automatic evaluation metrics (AEMs - such as BLEU) are typically used to measure quality im- provements in MT and quality improvements, in turn, are expected to lead to higher levels of satisfaction among post-editors. However, some AEMs have been shown not to correlate well with human evaluation of quality (Tatsu- mi, 2009), and although automatic metrics measuring edit distance such as Human- mediated Translation Edit Rate (HTER) (Snover et al., 2006) have better correlations with human judgements (Snover et al., 2009), evaluations with real users are often necessary …

Content truncated to 500 words