Evaluation of Machine-Translated User Generated Content: A pilot study based on User Ratings

Evaluation of Machine-Translated User Generated Content: A pilot study based on User Ratings Linda Mitchell SALIS Dublin City University Ballymun, Dublin 9, Ireland linda.mitchell7@mail.dcu.ie Johann Roturier Symantec SES EMEA Research Ballycoolin Business Park Blanchardstown, Dublin 15, Ireland johann_roturier@symantec.com Abstract This paper presents the results of an ex- perimental pilot user study, focusing on the evaluation of machine-translated us- er-generated content by users of an on- line community forum and how those us- ers interact with the MT content that is presented to them. Preliminary results show that ratings are very difficult to ob- tain, that a low percentage of posts (21%) was rated, that users need to be well in- formed about their task and that there is a weak correlation between the length of the post (number of words) and its com- prehensibility. 1 Introduction This study follows up on the work described in Roturier and Bensadoun (2011), in which four machine translation systems were compared in order to evaluate their suitability in translating user-generated content. In the present study, the objective is different since feedback on machine- translated content is solicited from actual users of an existing community forum (rather than using linguists or bilingual technical support agents). Thus, an additional objective is to analyse how users interact with the MT content presented to them. This paper is divided into four parts: in Section 2, related work is briefly discussed. In Section 3, the experimental design of this study is presented, while in Section 4 preliminary re- sults are reported. In Section 5, we make some conclusions and outline possible future work. © 2012 European Association for Machine Translation. 2 Related Work The machine-translation of user-generated con- tent has been identified as being potentially use- ful to allow communication between various user groups that do not share a common language (Flournoy and Rueppel, 2010). Indeed, it was announced in 2010 that TripAdvisor would be using Language Weaver-powered translations to make hotel reviews available in multiple lan- guages 1 . More recently, Facebook announced they would be using Microsoft’s Bing Translate for Page content2. Recent research work has also been performed in this area, including Roturier and Bensadoun (2011) and Banerjee et al. (2011). However, no study has focused on how machine-translated content would be received in- context by existing users of a forum community. 3 Experimental Design The current German Norton Community forum3 is composed of multiple sections, known as “boards”. We decided to create a specific board, where machine-translated content would be pub- lished4. In the introduction to this board, it was explained to users that this board is used to show machine translated posts (from English into German). A user (Max_MÜ) was created; a ficti- tious “MT robot” whose name is used to post machine-translated content. Additionally, a de- 1 http://blogs.forrester.com/tim_walters/10-07-15- sdl_casts_vote_machine_translation_language_weave 2 r_acquisition https://www.facebook.com/photo.php?fbid=10150491 112449572&set=a.121044129571.125587.103814695 71&type=1 3 http://de.community.norton.com 4 http://de.community.norton.com/t5/L%C3%B6sung- nicht-gefunden/bd-p/Max Proceedings of the 16th EAMT Conference, 28-30 May 2012, Trento, Italy 61 scription was added to Max_MÜ’s profile5, in- troducing himself and explaining the …

Content truncated to 500 words