TrustRank: Inducing Trust in Automatic Translations via Ranking

Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pages 612–621, Uppsala, Sweden, 11-16 July 2010. c⃝2010 Association for Computational Linguistics TrustRank: Inducing Trust in Automatic Translations via Ranking Radu Soricut Language Weaver, Inc. 6060 Center Drive, Suite 150 Los Angeles, CA 90045 rsoricut@languageweaver.com Abdessamad Echihabi Language Weaver, Inc. 6060 Center Drive, Suite 150 Los Angeles, CA 90045 echihabi@languageweaver.com Abstract The adoption of Machine Translation tech- nology for commercial applications is hampered by the lack of trust associated with machine-translated output. In this pa- per, we describe TrustRank, an MT sys- tem enhanced with a capability to rank the quality of translation outputs from good to bad. This enables the user to set a quality threshold, granting the user control over the quality of the translations. We quantify the gains we obtain in trans- lation quality, and show that our solution works on a wide variety of domains and language pairs. 1 Introduction The accuracy of machine translation (MT) soft- ware has steadily increased over the last 20 years to achieve levels at which large-scale commercial applications of the technology have become feasi- ble. However, widespread adoption of MT tech- nology remains hampered by the lack of trust as- sociated with machine-translated output. This lack of trust is a normal reaction to the erratic trans- lation quality delivered by current state-of-the- art MT systems. Unfortunately, the lack of pre- dictable quality discourages the adoption of large- scale automatic translation solutions. Consider the case of a commercial enterprise that hosts reviews written by travellers on its web site. These reviews contain useful information about hotels, restaurants, attractions, etc. There is a large and continuous stream of reviews posted on this site, and the large majority is written in En- glish. In addition, there is a large set of potential customers who would prefer to have these reviews available in their (non-English) native languages. As such, this enterprise presents the perfect oppor- tunity for the deployment of a large-volume MT solution. However, travel reviews present specific challenges: the reviews tend to have poor spelling, loose grammar, and broad topics of discussion. The result is unpredictable levels of MT quality. This is undesirable for the commercial enterprise, who is not content to simply reach a broad audi- ence, but also wants to deliver a high-quality prod- uct to that audience. We propose the following solution. We develop TrustRank, an MT system enhanced with a ca- pability to rank the quality of translation outputs from good to bad. This enables the user to set a quality threshold, granting the user control over the quality of the translations that it employs in its product. With this enhancement, MT adop- tion stops being a binary should-we-or-shouldn’t- we question. Rather, each user can make a per- sonal trade-off between the scope and the quality of their product. 2 Related Work Work on automatic MT evaluation started with the idea of comparing automatic translations against human-produced references. Such comparisons are done either at lexical level …

Content truncated to 500 words