Semantic Label Smoothing for Sequence to Sequence Problems

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 4992–4998, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 4992 Semantic Label Smoothing for Sequence to Sequence Problems Michal Lukasik, Himanshu Jain, Aditya Krishna Menon, Seungyeon Kim, Srinadh Bhojanapalli, Felix Yu, Sanjiv Kumar Google Research mlukasik@google.com Abstract Label smoothing has been shown to be an ef- fective regularization strategy in classification, that prevents overfitting and helps in label de- noising. However, extending such methods directly to seq2seq settings, such as Machine Translation, is challenging: the large target output space of such problems makes it in- tractable to apply label smoothing over all pos- sible outputs. Most existing approaches for seq2seq settings either do token level smooth- ing, or smooth over sequences generated by randomly substituting tokens in the target se- quence. Unlike these works, in this paper, we propose a technique that smooths over well formed relevant sequences that not only have sufficient n-gram overlap with the target se- quence, but are also semantically similar. Our method shows a consistent and significant im- provement over the state-of-the-art techniques on different datasets. 1 Introduction Label smoothing is a regularization technique com- monly used in deep learning (Szegedy et al., 2016; Chorowski and Jaitly, 2017; Vaswani et al., 2017; Zoph et al., 2018; Real et al., 2018; Huang et al., 2019), that improves calibration (M¨uller et al., 2019) and helps in label de-noising (Lukasik et al., 2020a). Here, one smooths labels by introducing a prior in the label space (often just a uniform distri- bution) in order to prevent overly confident predic- tions and achieve better model calibration, both of which lead to better generalization. Given these benefits, it is natural to con- sider whether label smoothing can be applied to sequence-to-sequence (seq2seq) prediction tasks in Natural Language Processing. Here, inducing a label prior involves smoothing in sequence space. However, this is a challenging task because the output space is exponentially large for sequences, unlike the label space in standard classification. Previous works approached this challenge either by smoothing over individual tokens of the target se- quence, or by sampling a few nearby targets accord- ing to Hamming distance or BLEU score (Norouzi et al., 2016; Elbayad et al., 2018). These techniques however do not guarantee that the smoothed targets lie within the space of acceptable targets (i.e., the sampled new target may no longer be grammati- cally correct or even preserve semantic meaning). In this work, we propose a label smoothing ap- proach for seq2seq problems that overcomes this limitation. Given a large-scale corpus of valid se- quences, our approach selects a subset of sequences that are not only semantically similar to the target sequence, but also well formed. We achieve this using a pre-trained model to find semantically simi- lar sequences from the corpus, and then use BLEU scores to rerank the closest targets. We empirically show that this approach improves over competitive baselines on multiple machine translation tasks. 2 Related Works Token-level smoothing A popular approach …

Content truncated to 500 words