QRev: Machine Translation of User Reviews: What Influences the Translation Quality?

QRev: Machine Translation of User Reviews: What Influences the Translation Quality? Maja Popovi´c ADAPT Centre School of Computing Dublin City University, Ireland maja.popovic@adaptcentre.ie Abstract This project aims to identify the important aspects of translation quality of user re- views which will represent a starting point for developing better automatic MT met- rics and challenge test sets, and will be also helpful for developing MT systems for this genre. We work on two types of reviews: Amazon products and IMDb movies, written in English and translated into two closely related target languages, Croatian and Serbian. 1 Description Data sets used for MT research include mainly ”formal written text” (such as news) and ”for- mal speech” (such as TED talks). Recently, there has been an increase of interest in the translation of ”informal written text” which focuses on very noisy texts originating from sources like What- sApp, Twitter and Reddit. On the other hand, other types of ”informal written text” such as user re- views have not been investigated thoroughly, al- though they are important both from commercial and from a user perspective – user reviews of prod- ucts have become an important feature that many customers expect to find. This project focusses on user reviews in order to investigate which new challenges this ”mid-way” kind of text poses for current MT systems. The main goal is to identify important quality aspects for MT of user reviews which will enable: • development of appropriate automatic evalu- ation metrics; c⃝ 2020 The authors. This article is licensed under a Creative Commons 3.0 licence, no derivative works, attribution, CC- BY-ND. • design of test suites specialised for important factors; • definition of directions for improving MT systems. Although the focus of the project are user reviews translated into Serbian and Croatian (as a case in- volving mid-size less-resourced morphologically rich European languages), the proposed evaluation strategy is completely genre/domain/language in- dependent, so it can be applied to any genre, do- main and language pair. 2 Data sets We are working with two types of publicly avail- able user reviews: • IMDb movie reviews1 • Amazon product reviews2 3 MT systems The main goal of the project is to find the common aspects important for the translation quality, and not to evaluate or compare particular MT systems. We are currently analysing MT outputs3 of three on-line systems: Google Translate4, Bing5 and Amazon translate6. We are also developing our own system using publicly available data, which will be analysed in the later stages of the project. 1https://ai.stanford.edu/˜amaas/data/ sentiment/ 2http://jmcauley.ucsd.edu/data/amazon/ 3generated at the end of January 2020 4https://translate.google.com/ 5https://www.bing.com/translator 6https://aws.amazon.com/translate/ 4 Evaluation procedure Our evaluation procedure is based on comprehen- sibility and fidelity (adequacy) (Roturier and Ben- sadoun, 2011), and it is being carried out on the re- view level (not on the sentence level). It should be noted that comprehensibility is not fluency – a flu- ent text can be incomprehensible, and vice versa. The novelty of our procedure is asking the annota- tors to concentrate …

Content truncated to 500 words