Round-trip Translation: What Is It Good For?

Proceedings of the Australasian Language Technology Workshop 2005, pages 127–133, Sydney, Australia, December 2005. Round-Trip Translation: What Is It Good For? Harold Somers School of Informatics Manchester University Manchester, UK Harold.Somers@manchester.ac.uk Abstract This paper considers the popular but ques- tionable technique of ‘round-trip transla- tion’ (RTT) as a means of evaluating free on-line Machine Translation systems. Two experiments are reported, both relat- ing to common requirements of lay-users of MT on the web. In the first we see whether RTT can accurately predict the overall quality of the MT system. In the second, we ask whether RTT can predict the translatability of a given text. In both cases, we find RTT to be a poor predictor of quality, with high BLEU and F-scores for RTTs when the forward translation was poor. We discuss why this is the case, and conclude that, even if it seemed obvi- ous that RTT was good for nothing, at least we now have some tangible evi- dence. 1 Introduction Macklovitch (2001:27) talks of “the spectacular growth and pervasiveness of the World Wide Web” leading to a “democratization” of Machine Translation (MT) which has “profoundly trans- formed the MT business”. The availability of free on-line MT systems, since CompuServe’s initial experiments in 1994 (Flanagan, 1996) and then more significantly AltaVista’s collaboration with Systran from 1997 onwards (Yang and Lange, 1998), has indeed revolutionized the MT world, creating a whole new and significantly large com- munity of users, mostly with little or no knowledge or understanding of how MT works or even, in some cases, how language works. Such users are, nevertheless, keen to know how good MT output is, and frequently resort to the intuitive technique of ‘round-trip translation’ (RTT), or ‘back-and- forth translation’, in which they take a given text or sentence, have it translated into some foreign lan- guage by the MT system (the ‘forward translation’, henceforth FT), then have it translated back into the original language by the same system (the ‘back translation’, BT). Popular articles on MT by journalists and other lay-users all too frequently use this technique to ‘evaluate’ MT, with results which are, depending on your predisposition, hilarious or infuriating. A recent example is from the Biomedical Transla- tions website (Anon, 2003), where the author ex- plains the technique, and suggests that “In theory, the back translated English should match the origi- nal English.” Several garbled examples are then given, and the article concludes “Would you trust your surgeon using these instructions?” Another website recognizes the problem “Machine transla- tions can produce text that is garbled or hilariously inaccurate”, and suggests as a resolution “Test the precision of your translated text by sending a phrase on a round trip through the translation en- gine.” (Anon, 2005). The dangers of this approach have long been appreciated: for example Huang (1990), addressing the problem of evaluating output when you do not know the target language, describes it as the “seemingly most natural way” to evaluate a trans- lation, but quickly warns that …

Content truncated to 500 words