Sequence Length is a Domain: Length-based Overfitting in Transformer Models

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 8246–8257 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 8246 Sequence Length is a Domain: Length-based Overfitting in Transformer Models Dušan Variš and Ondˇrej Bojar Faculty of Mathematics and Physics, Charles University, Malostranské námˇestí 25, 118 00 Prague, Czechia {varis,bojar}@ufal.mff.cuni.cz Abstract Transformer-based sequence-to-sequence ar- chitectures, while achieving state-of-the-art re- sults on a large number of NLP tasks, can still suffer from overfitting during training. In prac- tice, this is usually countered either by apply- ing regularization methods (e.g. dropout, L2- regularization) or by providing huge amounts of training data. Additionally, Transformer and other architectures are known to struggle when generating very long sequences. For example, in machine translation, the neural- based systems perform worse on very long sequences when compared to the preceding phrase-based translation approaches (Koehn and Knowles, 2017). We present results which suggest that the is- sue might also be in the mismatch between the length distributions of the training and valida- tion data combined with the aforementioned tendency of the neural networks to overfit to the training data. We demonstrate on a simple string editing task and a machine translation task that the Transformer model performance drops significantly when facing sequences of length diverging from the length distribution in the training data. Additionally, we show that the observed drop in performance is due to the hypothesis length corresponding to the lengths seen by the model during training rather than the length of the input sequence. 1 Introduction Current state-of-the-art Transformer-based se- quence generation models, either fine-tuned for chosen downstream tasks (Devlin et al., 2019), or trained from scratch for specific tasks such as ma- chine translation (Vaswani et al., 2017) or speech recognition (Pham et al., 2019), more and more often achieve performance comparable to that of humans (Hassan et al., 2018; Popel et al., 2020; Nguyen et al., 2020). However, such models fre- quently require billions of trainable parameters together with huge amounts of data (billions of tokens) to reach such performance (Brown et al., 2020). The good performance on held-out test sets seems to confirm the good generalization power of these models, although the inherent strong biases, sometimes leading to the use of a foul and toxic language, preserving stereotypes, etc., are well ac- knowledged (Gehman et al., 2020). Brown et al. (2020) claim that their Transformer model is also capable of simple arithmetics, however, it is yet to be validated whether the model truly learns the arithmetic algorithms or simply encodes a lookup table for a subset of specific examples. In this paper, we argue that the assumed gen- eralization power of the current state-of-the-art Transformer-based language generators does not come from the architecture itself but rather from the sheer volume of training data and the model’s ability to exploit the similarities between the train- ing and validation data. We demonstrate how the Transformer-based sequence-to-sequence models fail when the target sequence lengths of the training and validation data do not match. …

Content truncated to 500 words