As Easy as 1, 2, 3: Behavioural Testing of NMT Systems for Numerical Translation

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 4711–4717 August 1–6, 2021. ©2021 Association for Computational Linguistics 4711 As Easy as 1, 2, 3: Behavioural Testing of NMT Systems for Numerical Translation Jun Wang,1 Chang Xu,1 Francisco Guzmán,2 Ahmed El-Kishky,3∗ Benjamin I. P. Rubinstein,1 Trevor Cohn1 1University of Melbourne, Australia 2Facebook AI, 3Twitter Cortex jun2@student.unimelb.edu.au {xu.c3,benjamin.rubinstein,trevor.cohn}@unimelb.edu.au fguzman@fb.com, aelkishky@twitter.com Abstract Mistranslated numbers have the potential to cause serious effects, such as financial loss or medical misinformation. In this work we develop comprehensive assessments of the ro- bustness of neural machine translation systems to numerical text via behavioural testing. We explore a variety of numerical translation capa- bilities a system is expected to exhibit and de- sign effective test examples to expose system underperformance. We find that numerical mistranslation is a general issue: major com- mercial systems and state-of-the-art research models fail on many of our test examples, for high- and low-resource languages. Our tests re- veal novel errors that have not previously been reported in NMT systems, to the best of our knowledge. Lastly, we discuss strategies to mitigate numerical mistranslation. 1 Introduction Just as neural machine translation (NMT) systems have achieved tremendous benchmark results, they have been proven brittle when faced with irregu- lar inputs such as noisy text (Belinkov and Bisk, 2018; Michel and Neubig, 2018) or adversarial in- puts (Cheng et al., 2020). Among such errors, mis- translation of numerical text constitutes a crucial but under-explored category that may have pro- found implications. For example, in the medical domain, mistranslating the number of confirmed cases of a contagious disease like COVID-19 may exacerbate public health misinformation. Numeri- cal errors made in financial document translation, e.g., an extra or omitted digit or decimal point, could lead to significant monetary loss. Surpris- ingly, we find that numerical mistranslation is a general issue faced by state-of-the-art NMT sys- tems, including commercial and research systems, ∗This work was conducted while author was working at Facebook AI Type Input Output Separators The distance 557,601.101 meters.is [De]: Die Entfernung beträgt 557.601.101 Meter. Numerals The total weight is two hundred and two kg. [Zh]: 总 重 量 为220公斤。 Digits The R0 of the dis- ease is 3.28. [Ne]: रोगको R0 28.२28 हो। Units There were 100.01 million cases world- wide. [Zh]: 全 世 界 有1.001 亿病 例。 (100.1 million) Table 1: Numerical errors discovered by our method when behavioural testing two popular commercial trans- lation systems using their public APIs. with evidence present across contexts: for both high and low resource languages, and for both close and distant languages. De facto standard metrics such as BLEU (Pa- pineni et al., 2002) may fail to flag a numerical translation error, which only contributes a very mi- nor penalty, as it is typically a single-token mis- translation. To facilitate the discovery of numer- ical errors made by NMT systems, we propose a black-box test method1 for assessing and de- bugging the numerical translation of NMT sys- tems in a systematic manner. Our method ex- tends …

Content truncated to 500 words