Agent and User-Generated Content and its Impact on Customer Support MT

Agent and User-Generated Content and its Impact on Customer Support MT Madalena Gonc¸alves* Marianna Buchicchio* Craig Stewart* Helena Moniz*†‡ Alon Lavie* *Unbabel †INESC-ID, Lisboa, Portugal ‡University of Lisbon, Lisboa, Portugal *{firstname.lastname}@unbabel.com Abstract This paper illustrates a new evaluation framework developed at Unbabel for mea- suring the quality of source language text and its effect on both Machine Translation (MT) and Human Post-Edition (PE) per- formed by non-professional post-editors. We examine both agent and user-generated content from the Customer Support do- main and propose that differentiating the two is crucial to obtaining high qual- ity translation output. Furthermore, we present results of initial experimentation with a new evaluation typology based on the Multidimensional Quality Metrics (MQM) Framework (Lommel et al., 2014), specifically tailored toward the evaluation of source language text. We show how the MQM Framework (Lommel et al., 2014) can be adapted to assess errors of mono- lingual source texts and demonstrate how very specific source errors propagate to the MT and PE targets. Finally, we illustrate how MT systems are not robust enough to handle specific types of source noise in the context of Customer Support data. 1 Introduction Unbabel’s Language Operations platform blends advanced artificial intelligence with humans in the loop, for fast, efficient, high-quality trans- lations that get smarter over time. The com- pany combines Machine Translation with Human Post-Edition performed by non-professional post- editors to translate Customer Support content in a © 2022 The authors. This article is licensed under a Creative Commons 3.0 licence, no derivative works, attribution, CC- BY-ND. variety of formats including emails and chat mes- sages. Customer Support is a highly unique do- main given that it involves bilateral communica- tion between Customer Support agents (‘agents’) and customers (‘users’), each with their own nu- anced discourse strategies and features. Notwithstanding, primary literature such as Nars et al. (2016), generally consider both sides of the interaction jointly, without regard for indepen- dent and differentiating factors. The fundamental differences can be characterized as follows: agents are call center employees who are usually non- native speakers of English, which has generally evolved as the ‘lingua franca’ of Customer Support and the primary source language translated at Un- babel. Given that English is usually not the agent’s first language, it is common to observe elements of language transfer (where the grammar rules of their native language are transferred to the English language) and other linguistic errors, more com- monly the addition and omission of prepositions and articles, as also mentioned in Lee and Seneff (2008) and Rozovskaya and Roth (2010). Accord- ing to Sinha et al. (2009), a person’s experience and knowledge of their mother tongue will most definitely interfere with the learning of a second language, thus creating errors of different nature. The interaction established by agents is some- what controlled because they are usually follow- ing a particular protocol for communication pre- scribed by their company. This might include re- sponse templates, branding and fixed terminology which discourages stylistic variance and can often result …

Content truncated to 500 words