Noise Isn’t Always Negative: Countering Exposure Bias in Sequence-to-Sequence Inflection Models

Proceedings of the 28th International Conference on Computational Linguistics, pages 2837–2846 Barcelona, Spain (Online), December 8-13, 2020 2837 Noise Isn’t Always Negative: Countering Exposure Bias in Sequence-to-Sequence Inflection Models Garrett Nicolai Department of Linguistics University of British Columbia garrett.nicolai@ubc.ca Miikka Silfverberg Department of Linguistics University of British Columbia msilfver@ubc.ca Abstract Morphological inflection, like many sequence-to-sequence tasks, sees great performance from recurrent neural architectures when data is plentiful, but performance falls off sharply in lower- data settings. We investigate one aspect of neural seq2seq models that we hypothesize contributes to overfitting - teacher forcing. By creating different training and test conditions, exposure bias increases the likelihood that a system too closely models its training data. Experiments show that teacher-forced models struggle to recover when they enter unknown territory. However, a simple modification to the training algorithm to more closely mimic test conditions creates models that are better able to generalize to unseen environments. 1 Introduction Morphological inflection has gained substantial interest in recent years due to a number of shared tasks focusing on morphology learning (Cotterell et al., 2016; Cotterell et al., 2017; Cotterell et al., 2018; McCarthy et al., 2019; Vylomova et al., 2020; Kann et al., 2020). Systems for inflection are trained on a sequence of pairs: {Lemma, MorphoSyntactic Descriptor (MSD)}, free of context, and must produce as output an inflected word form. For example, given the input pair {run, V.PTCP;PRS}, a successful system should produce the present participial form: running. The shared tasks have illuminated several idiosyncrasies of inflection. Although the state-of-the-art methods have been inspired by Neural Machine Translation (NMT), inflection is, in many ways, a more straightforward task. Often, a majority of characters can be copied directly from input to output, and reordering of tokens is not necessary to the extent that it is in translation. Thus, systems with copy- mechanisms (Makarov et al., 2017), and hard, monotonic attention (Aharoni and Goldberg, 2017) tend to perform well, even when data is scarce. Recurrent neural architectures require that the output from a previous time step be fed back into the model. During training, existing systems all use a variant of teacher forcing, which feeds gold-standard tokens into the model at time t + 1. This methodology differs significantly from how the model pro- gresses at test-time, where silver tokens must be used. This problem has been dubbed exposure bias by Wiseman and Rush (2016). We hypothesize that exposure bias can lead models to overfit the training data, particularly in low-resource settings. In this paper, we compare teacher forcing to silver label propagation, or student forcing, where model predictions are fed into the decoder during training instead of gold standard labels. Our experiments on a geographically and linguistically diverse set of 10 languages show that student forcing typically outperforms teacher forcing in a low-resource setting. Our analysis of the results suggests that teacher forced models are, indeed, overfitting the training data by ignoring input characters at test time. We note substantial gains when student forcing is applied to …

Content truncated to 500 words