Measuring the behavioral impact of machine translation quality improvements with A/B testing

Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2295–2299, Austin, Texas, November 1-5, 2016. c⃝2016 Association for Computational Linguistics Measuring the behavioral impact of machine translation quality improvements with A/B testing Benjamin Russell and Duncan Gillespie Etsy {brussell, dgillespie}@etsy.com Abstract In this paper we discuss a process for quan- tifying the behavioral impact of a domain- customized machine translation system de- ployed on a large-scale e-commerce platform. We discuss several machine translation sys- tems that we trained using aligned text from product listing descriptions written in mul- tiple languages. We document the qual- ity improvements of these systems as mea- sured through automated quality measures and crowdsourced human quality assessments. We then measure the effect of these quality im- provements on user behavior using an au- tomated A/B testing framework. Through testing we observed an increase in key e- commerce metrics, including a significant in- crease in purchases. 1 Introduction Quality evaluation is an essential task when train- ing a machine translation (MT) system. While au- tomatic evaluation methods like BLEU (Papineni et al., 2002) can be useful for estimating translation quality, a higher score is no guarantee of quality improvement (Callison-Burch et al., 2006). Previ- ous studies (e.g. Coughlin, 2003) have compared human evaluations of MT to metrics like BLEU and found close correspondence between the two. Koehn (2004) argued that relatively small differ- ences in BLEU can indicate significant MT qual- ity differences and suggested that human evaluation, the traditional alternative to automated metrics like BLEU, is therefore unnecessarily time-consuming and costly. Callison-Burch (2009) explored the use of crowdsourcing platforms for evaluating MT qual- ity, with good results. However, we are not aware of any research that investigates the effect of im- proved MT on human behavior. In a commercial application, like an e-commerce platform, it is de- sirable to have a high degree of confidence in the material effect of MT quality differences: any MT system change should positively impact user experi- ences. Etsy is an online marketplace for handmade and vintage items, with over 40 million active listings and a community of buyers and sellers located around the world. Visitors can use MT to translate the text of product descriptions, product reviews, and private messages, making it possible for mem- bers to communicate effectively with one another, even when they speak different languages. These multilingual interactions facilitated by MT, such as reading nonnative listing descriptions or conversing with a foreign seller, are integral to the user experi- ence. However, due to the unique nature of the products available in the marketplace, a generic third party MT system1 often falls short when translating user- generated content. One challenging lexical item is “clutch.” A generic engine, trained on commonly available parallel text, translates clutch as an “auto- motive clutch.” In this marketplace, however, clutch almost always means “purse.” A mistake like this is problematic: a user who sees this incorrect machine translation may lose confidence in that listing and possibly in the marketplace …

Content truncated to 500 words