Human Effort and Machine Learnability in Computer Aided Translation

Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1225–1236, October 25-29, 2014, Doha, Qatar. c⃝2014 Association for Computational Linguistics Human Effort and Machine Learnability in Computer Aided Translation Spence Green, Sida Wang, Jason Chuang,* Jeffrey Heer,* Sebastian Schuster, and Christopher D. Manning Computer Science Department, Stanford University {spenceg,sidaw,sebschu,manning}@stanford.edu *Computer Science Department, University of Washington {jcchuang,jheer}@uw.edu Abstract Analyses of computer aided translation typi- cally focus on either frontend interfaces and human effort, or backend translation and machine learnability of corrections. How- ever, this distinction is artificial in prac- tice since the frontend and backend must work in concert. We present the first holis- tic, quantitative evaluation of these issues by contrasting two assistive modes: post- editing and interactive machine translation (MT). We describe a new translator inter- face, extensive modifications to a phrase- based MT system, and a novel objective function for re-tuning to human correc- tions. Evaluation with professional bilin- gual translators shows that post-edit is faster than interactive at the cost of translation quality for French-English and English- German. However, re-tuning the MT sys- tem to interactive output leads to larger, sta- tistically significant reductions in HTER versus re-tuning to post-edit. Analysis shows that tuning directly to HTER results in fine-grained corrections to subsequent machine output. 1 Introduction The goal of machine translation has always been to reduce human effort, whether by partial assistance or by outright replacement. However, preoccupa- tion with the latter—fully automatic translation—at the exclusion of the former has been a feature of the research community since its first nascent steps in the 1950s. Pessimistic about progress during that decade and future prospects, Bar-Hillel (1960, p.3) argued that more attention should be paid to a “machine-post-editor partnership,” whose decisive problem is “the region of optimality in the contin- uum of possible divisions of labor.” Today, with human-quality, fully automatic machine translation (MT) elusive still, that decades-old recommenda- tion remains current. This paper is the first to look at both sides of the partnership in a single user study. We compare two common flavors of machine-assisted transla- tion: post-editing and interactive MT. We analyze professional, bilingual translators working in both modes, looking first at user productivity. Does the additional machine assistance available in the inter- active mode affect translation time and/or quality? Then we turn to the machine side of the part- nership. The user study results in corrections to the baseline MT output. Do these corrections help the MT system, and can it learn from them quickly enough to help the user? We perform a re-tuning experiment in which we directly optimize human Translation Edit Rate (HTER), which correlates highly with human judgments of fluency and ade- quacy (Snover et al., 2006). It is also an intuitive measure of human effort, making fine distinctions between 0 (no editing) and 1 (complete rewrite). We designed a new user interface (UI) for the experiment. The interface places demands on the MT backend—not the other way around. The most significant new MT system …

Content truncated to 500 words