Jam or Cream First? Modeling Ambiguity in Neural Machine Translation with SCONES

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4950 - 4961 July 10-15, 2022 ©2022 Association for Computational Linguistics Jam or Cream First?1 Modeling Ambiguity in Neural Machine Translation with SCONES Felix Stahlberg and Shankar Kumar Google Research {fstahlberg,shankarkumar}@google.com Abstract The softmax layer in neural machine transla- tion is designed to model the distribution over mutually exclusive tokens. Machine transla- tion, however, is intrinsically uncertain: the same source sentence can have multiple se- mantically equivalent translations. Therefore, we propose to replace the softmax activation with a multi-label classification layer that can model ambiguity more effectively. We call our loss function Single-label Contrastive Objec- tive for Non-Exclusive Sequences (SCONES). We show that the multi-label output layer can still be trained on single reference training data using the SCONES loss function. SCONES yields consistent BLEU score gains across six translation directions, particularly for medium- resource language pairs and small beam sizes. By using smaller beam sizes we can speed up inference by a factor of 3.9x and still match or improve the BLEU score obtained using softmax. Furthermore, we demonstrate that SCONES can be used to train NMT mod- els that assign the highest probability to ade- quate translations, thus mitigating the “beam search curse”. Additional experiments on syn- thetic language pairs with varying levels of un- certainty suggest that the improvements from SCONES can be attributed to better handling of ambiguity. 1 Introduction Conventional neural machine translation (NMT) models learn the probability P(y|x) of the target sentence y given the source sentence x (Kalchbren- ner and Blunsom, 2013; Sutskever et al., 2014). This framework implies that there is a single best translation for a given source sentence: if there were multiple valid translations y1 and y2 they would need to share probability mass (e.g. P(y1|x) = 0.5 and P(y2|x) = 0.5), but such 1https://en.wikipedia.org/wiki/Cream_ tea#Variations a distribution could also represent model uncer- tainty, i.e. the case when either y1 or y2 are correct translations. Therefore, learning a single distri- bution over all target language sentences does not allow the model to naturally express intrinsic uncer- tainty2 (Padó et al., 2009; Dreyer and Marcu, 2012; Ott et al., 2018; Stahlberg et al., 2022), the nature of the translation task to allow multiple seman- tically equivalent translations for a given source sentence. A single distribution over all sequences represents uncertainty by assigning probabilities, but it cannot distinguish between different kinds of uncertainty (e.g. model uncertainty versus intrinsic uncertainty). Therefore, in this work we frame ma- chine translation as a multi-label classification task (Tsoumakas and Katakis, 2007; Zhang and Zhou, 2014). Rather than learning a single distribu- tion P(y|x) over all target sentences y for a source sentence x, we learn binary classifiers for each sen- tence pair (x, y) that indicate whether or not y is a valid translation of x. In this framework, intrin- sic uncertainty can be represented by setting the probabilities of two (or more) correct translations y1 …

Content truncated to 500 words