Quality Fit for Purpose: Building Business Critical Errors Test Suites

Quality Fit for Purpose: Building Business Critical Errors Test Suites Mariana Cabec¸a†γι∗ Marianna Buchicchio‡ι∗ Madalena Gonc¸alves◦ι∗ Christine Maroti‡ι∗ Jo˜ao Godinho◦ι∗ Pedro Coelho◦ι∗ Helena Monizαγι Alon Lavie ◦ι ιUnbabel γINESC-ID † marianacabeca14@inesc-id.pt ◦{firstname.lastname}@unbabel.com ‡ {firstname}@unbabel.com α helena.moniz@inesc-id.pt Abstract This paper illustrates a new methodology based on Test Suites (Avramidis et al., 2018) with focus on Business Critical Er- rors (BCEs) (Stewart et al., 2022) to eval- uate the output of Machine Translation (MT) and Quality Estimation (QE) sys- tems. We demonstrate the value of re- lying on semi-automatic evaluation done through scalable BCE-focused Test Suites to monitor both MT and QE systems’ per- formance for 8 language pairs (LPs) and a total of 4 error categories. This approach allows us to not only track the impact of new features and implementations in a real business environment, but also to identify strengths and weaknesses in models re- garding different error types, and subse- quently know what to improve henceforth. 1 Introduction Unbabel’s Language Operations platform blends advanced artificial intelligence with humans in the loop for fast, efficient and high-quality transla- tions that get smarter over time. The company combines Machine Translation with Human Post- Edition performed by experienced post-editors to translate a variety of content, ranging from Cus- tomer Support to Marketing. MT and Quality Evaluation are at the core of Unbabel’s business, as the main focus is to provide high-quality trans- lations regardless of the use case or content type. Both MT and QE systems have been continu- ously improving and overcoming existing limita- ∗These authors contributed equally. ∗© 2023 The authors. This article is licensed under a Creative Commons 4.0 licence, no derivative works, attribution, CC- BY-ND. tions throughout the years. As a result, the need to evaluate their outputs’ accuracy and overall per- formance in error detection grows along with this development process, especially in a business en- vironment where the need to deliver high quality translations without critical errors is paramount. The evaluation of MT outputs can be gener- ally done by following either manual quality as- sessment procedures with error annotations (such as the Multidimensional Quality Metrics (MQM) Framework (Lommel et al., 2014)), or automati- cally by relying on metrics such as BLEU (Pap- ineni et al., 2002), METEOR (Banerjee and Lavie, 2005), and COMET (Rei et al., 2020). In the same way that MT evaluation and com- plementary metrics are crucial to achieve outputs with better quality, so is the need to evaluate the precision and accuracy of QE systems. To this end, QE systems are oftentimes evaluated against gold annotated data by the Pearson correlation score (e.g., Fonseca et al. (2019)) and by com- puting the Matthews correlation coefficient (MCC, (Matthews, 1975)). The main focus of this paper is to overcome the shortcomings of both manual and automatic MT and QE evaluation methodologies in a ‘real-life’ business environment. We are able to achieve this through a semi-automatic approach that relies on MT Test Suites (Avramidis et al., 2018) in a pro- duction setting. The MT …

Content truncated to 500 words