Rethinking Perturbations in Encoder-Decoders for Fast Training

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5767–5780 June 6–11, 2021. ©2021 Association for Computational Linguistics 5767 Rethinking Perturbations in Encoder-Decoders for Fast Training Sho Takase Tokyo Institute of Technology sho.takase@nlp.c.titech.ac.jp Shun Kiyono RIKEN / Tohoku University shun.kiyono@riken.jp Abstract We often use perturbations to regularize neural models. For neural encoder-decoders, previous studies applied the scheduled sampling (Bengio et al., 2015) and adversarial perturbations (Sato et al., 2019) as perturba- tions but these methods require considerable computational time. Thus, this study addresses the question of whether these approaches are efficient enough for training time. We compare several perturbations in sequence-to-sequence problems with respect to computational time. Experimental results show that the simple techniques such as word dropout (Gal and Ghahramani, 2016) and random replacement of input tokens achieve comparable (or better) scores to the recently proposed perturba- tions, even though these simple methods are faster. Our code is publicly available at https://github.com/takase/rethink_perturbations. 1 Introduction Recent advances in neural encoder-decoders have driven tremendous success for sequence- to-sequence problems including machine transla- tion (Sutskever et al., 2014), summarization (Rush et al., 2015), and grammatical error correction (GEC) (Ji et al., 2017). Since neural models can be too powerful, previous studies have proposed vari- ous regularization methods to avoid over-fitting. To regularize neural models, we often apply a perturbation (Goodfellow et al., 2015; Miyato et al., 2017), which is a small difference from a correct input. During the training process, we force the model to output the correct labels for both per- turbed inputs and unmodified inputs. In sequence- to-sequence problems, existing studies regard the following as perturbed inputs: (1) sequences con- taining tokens replaced from correct ones (Bengio et al., 2015; Cheng et al., 2019), (2) embeddings injected small differences (Sato et al., 2019). For example, Bengio et al. (2015) proposed the sched- uled sampling that samples a token from the output probability distribution of a decoder and uses it as a perturbed input for the decoder. Sato et al. (2019) applied an adversarial perturbation, which signif- icantly increases the loss value of a model, to the embedding spaces of neural encoder-decoders. Those studies reported that their methods are ef- fective to construct robust encoder-decoders. How- ever, their methods are much slower than the train- ing without using such perturbations because they require at least one forward computation to obtain the perturbation. In fact, we need to run the de- coder the same times as the required number of perturbations in the scheduled sampling (Bengio et al., 2015). For adversarial perturbations (Sato et al., 2019), we have to compute the backpropa- gation in addition to forward computation because we use gradients to obtain perturbations. Those properties seriously affect the training budget. For example, it costs approximately 1,800 USD for each run when we train Transformer (big) with adversarial perturbations (Sato et al., 2019) on the widely used WMT English-German training set in AWS EC21. Most studies conduct multiple runs for …

Content truncated to 500 words