Using the ACCEPT framework to conduct an online community-based translation evaluation study

Using the ACCEPT framework to conduct an online community-based translation evaluation study Linda Mitchell Symantec Ltd. Ballycoolin Business Park Dublin 15, Ireland linda mitchell@symantec.com Johann Roturier Symantec Ltd. Ballycoolin Business Park Dublin 15, Ireland johann roturier@symantec.com David Silva Symantec Ltd. Ballycoolin Business Park Dublin 15, Ireland david silva@symantec.com Abstract This paper presents how a novel evalua- tion framework was used to collect trans- lation ratings thanks to users of an on- line German-speaking support community in the IT domain. Using an innovative data collection approach and mechanism, this paper shows that segment-level ratings can be collected in an effective manner. The collection mechanism leverages the AC- CEPT evaluation framework which allows data collection to be triggered from online environments in which community users interact on a regular basis. 1 Introduction While machine translation is becoming ubiquitous in making Web content accessible to users who do not necessarily understand the language in which this content was first authored, some doubts remain about the ability of machine translation systems to generate content that is sufficiently fluent to be easily understood. Collecting translation evalua- tion ratings or judgments is often done in dedicated evaluation environments which fail to take into account any ecological validity requirements that are inherent to user-focused settings where trans- lated content is made available. The approach pre- sented in this paper addresses this issue by leverag- ing the evaluation framework provided by the AC- CEPT project and by designing an evaluation task aimed at being as self-contained, self-explanatory, intuitive and engaging as possible for the target population of raters. The paper is organized as fol- lows: Section 2 briefly reviews existing evaluation c⃝ 2014 The authors. This article is licensed under a Creative Commons 3.0 licence, no derivative works, attribution, CC- BY-ND. frameworks Section 3 focuses on the setup of an evaluation task that maximizes user engagement. Section 4 presents some statistics on the evaluation data collected during a 4-week timeframe. Finally, Section 5 contains some preliminary conclusions and suggestions for future work. 2 Related work Numerous (machine) translation evaluation sys- tems exist. These systems can be grouped into four categories: standalone desktop-based sys- tems, Web-based dedicated systems, Web-based generic systems and Web-based hybrid systems. The first type of system (standalone desktop-based system such as Costa (Chatzitheodorou, 2013) is not relevant for studies involving online com- munity members because it is inconvenient to ask users to install an application to provide rat- ings. Web-based dedicated systems such as Ap- praise (Federmann, 2012) or the Dynamic Quality Framework (DQF) Tools1 may be useful to col- lect judgments from well-known contributors but they are not suited to collect genuine user feedback (mainly because they require a separate account creation and login process which can be cum- bersome for users whose first priority is to con- sume content rather than evaluate content). Web- based generic systems such as Amazon Mechani- cal Turk (Callison-Burch, 2009) or Crowdflower2 suffer from the same problem as dedicated sys- tems. While they can be useful in generating …

Content truncated to 500 words