Organic.Lingua - Demonstrating the Potential of a multilingual Web portal for Sustainable Agricultural & Environmental Education

User Evaluation of Interactive Machine Translation Systems Vicent Alabau, Luis A. Leiva, Daniel Ortiz-Mart´ınez, Francisco Casacuberta ITI/DSIC – Universitat Polit`ecnica de Val`encia {valabau,luileito,dortiz,fcn}@{iti,dsic}.upv.es Abstract Recent developments in search algorithms and software architecture have enabled multi-user web-based prototypes for Inter- active Machine Translation (IMT), a tech- nology that aims to assist, rather than re- place, the human translator. Surprisingly, formal human evaluations of IMT systems are highly scarce in the literature. To this regard, we discuss experiences gained while testing IMT systems. We report the lessons learned from two user evalua- tions. Our results can provide researchers and practitioners with several guidelines towards the design of on-line IMT tools. 1 Introduction Research in machine translation (MT) aims to de- velop computer systems which are able to translate documents without human intervention. However, current translation technology has not been able to deliver full automated error-free translations. Typ- ical solutions to improve the quality of an MT sys- tem require manual post-editing. This serial pro- cess does not allow integrating the knowledge of the human translator into the system decisions. One alternative to take advantage of the ex- isting MT technologies is to apply the so-called interactive machine translation (IMT) paradigm (Langlais et al., 2002). The IMT paradigm adapts data driven MT techniques for its use in collab- oration with human translators. Following these ideas, Barrachina et al. (2009) proposed a new ap- proach to IMT, in which fully-fledged statistical MT systems are used to produce full target sen- tences hypotheses, or portions thereof, which can be accepted or amended by a human translator. Each corrected text segment is then used by the MT system as additional information to achieve improved suggestions. Figure 1 shows a minimal IMT session example. c⃝ 2012 European Association for Machine Translation. source: Para ver la lista de recursos reference: To view a listing of resources suggestion s To view the resources list interaction kp To view a s listing of resources accept p To view a listing of resources Figure 1: An IMT session example, using only 1 key stroke (k) to achieve the reference sentence. Notice that the user submits partial sentences (p) to the system, which tries to complete them (s). Following the IMT paradigm, recent develop- ments in search algorithms and software architec- ture have allowed multi-user web-based transla- tion prototypes. These systems have grown in fea- tures, e.g., allowing advanced multimodal interac- tion, which have also added extra complexity to the prototypes. Then, their effectiveness should be tested with respect to technology dissemination. While pure data-driven evaluations have already shown that IMT is a promising technology (Bar- rachina et al., 2009), surprisingly, formal human evaluations are highly scarce in the literature. In this paper, we describe our experiences eval- uating two IMT prototypes with real users: an initial, advanced version and a simplified but im- proved version. Our results identify important design issues, which open a discussion regarding how IMT systems should be deployed. 2 Related Work Langlais et al. (2002) performed a …

Content truncated to 500 words