Evaluating User Preferences in Machine Translation Using Conjoint Analysis

Evaluating User Preferences in Machine Translation Using Conjoint Analysis Katrin Kirchhoff Department of Electrical Engineering University of Washington Seattle, WA, USA katrin@ee.washington.edu Daniel Capurro, Anne Turner Department of Medical Education and Biomedical Informatics University of Washington Seattle, WA, USA dcapurro@u.washington.edu amturner@u.washington.edu Abstract In spite of much ongoing research on ma- chine translation evaluation there is little quantitative work that directly measures users’ intuitive or emotional preferences regarding different types of machine trans- lation errors. However, the elicitation and modeling of user preferences is an im- portant prerequisite for future research on user adaptation and customization of ma- chine translation engines. In this paper we explore the use of conjoint analysis as a formal quantitative framework to gain in- sight into users’ relative preferences for different translation error types. Using English-Spanish as the translation direc- tion we conduct a crowd-sourced conjoint analysis study and obtain utility values for individual error types. Our results indicate that word order errors are clearly the most dispreferred error type, followed by word sense, morphological, and function word errors. 1 Introduction Current work in machine translation (MT) evalu- ation research falls into three different categories: automatic evaluation, human evaluation, and em- bedded application evaluation. Much effort has focused on the first category, i.e. on designing eval- uation metrics that can be computed automatically for the purpose of system tuning and development. These include e.g. BLEU (Papineni et al., 2002), position-independent word error rate (PER), ME- TEOR (Lavie and Agarwal, 2007), or translation error rate (TER) (Snover et al., 2006). Human c⃝ 2012 European Association for Machine Translation. evaluation (see (Denkowskie and Lavie, 2010) for a recent overview) typically involves rating trans- lation output with respect to fluency and adequacy (LDC, 2005), or directly comparing and ranking two or more translation outputs (Callison-Burch et al., 2007). All of these evaluation techniques pro- vide a global assessment of overall translation per- formance without regard to different error types. More fine-grained analyses of individual MT er- rors often include manual or (semi-) automatic er- ror annotation to gain insights into the strengths and weaknesses of MT engines (Vilar et al., 2006; Popovic and Ney, 2011; Condon et al., 2010; Far- reus et al., 2012). There have also been studies of how MT errors influence the work of post-editors with respect to productivity, speed, etc. (Krings, 2001; O’Brien, 2011) or the performance of back- end applications like information retrieval (Parton and McKeown, 2010). In contrast to this line of research, there is surprisingly little work that directly investigates which types of errors are intuitively the most dis- liked by users of machine translation. Although there is ample anecdotal evidence of users’ reac- tions to machine translation, it is difficult to find formal, quantitative studies of how users perceive the severity of different translation errors and what trade-offs they would make between different er- rors if they were given a choice. User prefer- ences might sometimes diverge strongly from the system development directions suggested by auto- matic evaluation procedures. Most automatic …

Content truncated to 500 words