Rare but Severe Neural Machine Translation Errors Induced by Minimal Deletion: An Empirical Study on Chinese and English

Proceedings of the 29th International Conference on Computational Linguistics, pages 5175–5180 October 12–17, 2022. 5175 Rare but Severe Neural Machine Translation Errors Induced by Minimal Deletion: An Empirical Study on Chinese and English Ruikang Shi and Alvin Grissom II and Duc Minh Trinh Haverford College Haverford, PA, USA Abstract We examine the inducement of rare but severe errors in English-Chinese and Chinese-English in-domain neural machine translation by mini- mal deletion of the source text with character- based models. By deleting a single character, we can induce severe translation errors. We categorize these errors and compare the results of deleting single characters and single words. We also examine the effect of training data size on the number and types of pathological cases induced by these minimal perturbations, find- ing significant variation. We find that deleting a word hurts overall translation score more than deleting a character, but certain errors are more likely to occur when deleting characters, with language direction also influencing the effect. 1 Introduction Pathological machine translation (MT) errors have been a problem since the field’s inception, and they have been analyzed and categorized in the context of both statistical (SMT) and neural machine trans- lation (NMT). Recent work examines pathologies in NLP models on classification problems: cases in which the models make wildly inaccurate pre- dictions, often confidently, when input tokens are removed (Feng et al., 2018). Identifying these en- riches our understanding of neural models and their points of failure. MT pathologies take the form of severe translation errors, the worst being halluci- nations (Lee et al., 2019). These rare errors are difficult to study precisely because they are rare. In this paper, we examine severe errors induced by minimal deletions by automatically extracting translations with severe errors and manually cate- gorizing them. Previous work taxonomizes SMT errors (Vilar et al., 2006) and analyzes their effects on trans- lation quality (Federico et al., 2014). More re- cently, Guerreiro et al. (2022) propose a taxonomy of MT pathologies, of which hallucinations are a category.1 They note the shortcomings of current automatic detection methods, e.g., those based on quality estimation and heuristics, and look for crit- ical errors in naturalistic settings. They also pro- pose DEHALLUCINATOR, which flags problematic translations and replaces them with re-ranking. Other work on Chinese-English (Zh-En) SMT ex- amines tense errors caused by incorrectly translat- ing 了 (le) (Liu et al., 2011) and syntactic failures caused by 的 (de). More recent work uses input perturbation to argue that NMT models, including those based on transformers (Vaswani et al., 2017), are brittle: Belinkov and Bisk (2018) examine the effect on NMT systems of several kinds of random- ized perturbations by adding tokens, and Niu et al. (2020) study subword regularization to increase robustness to randomized perturbations. Raunak et al. (2021) argue that memorized training exam- ples are more likely to hallucinate, and Voita et al. (2020) examine the contribution of source and tar- get tokens to errors. Also related, Sun et al. (2020) suggest that BERT …

Content truncated to 500 words