On Long-Tailed Phenomena in Neural Machine Translation

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3088–3095 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 3088 On Long-Tailed Phenomena in Neural Machine Translation Vikas Raunak1 Siddharth Dalmia1 Vivek Gupta2 Florian Metze1 1 Carnegie Mellon University, USA 2 University of Utah, USA {vraunak,sdalmia,fmetze}@cs.cmu.edu vgupta@cs.utah.edu Abstract State-of-the-art Neural Machine Translation (NMT) models struggle with generating low- frequency tokens, tackling which remains a major challenge. The analysis of long-tailed phenomena in the context of structured predic- tion tasks is further hindered by the added com- plexities of search during inference. In this work, we quantitatively characterize such long- tailed phenomena at two levels of abstraction, namely, token classification and sequence gen- eration. We propose a new loss function, the Anti-Focal loss, to better adapt model train- ing to the structural dependencies of condi- tional text generation by incorporating the in- ductive biases of beam search in the training process. We show the efficacy of the proposed technique on a number of Machine Translation (MT) datasets, demonstrating that it leads to significant gains over cross-entropy across dif- ferent language pairs, especially on the gen- eration of low-frequency words. We have re- leased the code to reproduce our results.1 1 Introduction Autoregressive sequence to sequence (seq2seq) models such as Transformers (Vaswani et al., 2017) are trained to maximize the log-likelihood of the target sequence, conditioned on the input sequence. Furthermore, approximate inference (search) is typically done using the beam search algorithm (Reddy, 1988), which allows for a controlled explo- ration of the exponential search space. However, seq2seq models (or structured prediction models in general) suffer from a discrepancy between token level classification during learning and sequence level inference during search. This discrepancy also manifests itself in the form of the curse of sentence length i.e. the models’ proclivity to gen- erate shorter sentences during inference, which The first author is now a researcher at Microsoft, USA. 1https://github.com/vyraun/long-tailed has received considerable attention in the literature (Pouget-Abadie et al., 2014; Murray and Chiang, 2018). In this work, we focus on how to better model long-tailed phenomena, i.e. predicting the long-tail of low-frequency words/tokens (Zhao and Marcus, 2012), in seq2seq models, on the task of Neural Machine Translation (NMT). Essentially, there are two mechanisms by which tokens with low fre- quency receive lower probabilities during predic- tion: firstly, the norms of the embeddings of low frequency tokens are smaller, which means that during the dot-product based softmax operation to generate a probability distribution over the vocab- ulary, they receive less probability. This has been well known in Image Classification (Kang et al., 2020) and Neural Language Models (Demeter et al., 2020). Since NMT shares the same dot-product softmax operation, we observe that the same phe- nomenon holds true for NMT as well. For example, we observe a Spearman’s Rank Correlation of 0.43 between the norms of the token embeddings and their frequency, when a standard transformer model is trained on the IWSLT-14 De-En dataset (more de- tails in section 2). Secondly, for …

Content truncated to 500 words