TeaForN: Teacher-Forcing with N-grams

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 8704–8717, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 8704 TeaForN: Teacher-Forcing with N-grams Sebastian Goodman Google Research Venice, CA 90291 seabass@google.com Nan Ding Google Research Venice, CA 90291 dingnan@google.com Radu Soricut Google Research Venice, CA 90291 rsoricut@google.com Abstract Sequence generation models trained with teacher-forcing suffer from issues related to ex- posure bias and lack of differentiability across timesteps. Our proposed method, Teacher- Forcing with N-grams (TeaForN), addresses both these problems directly, through the use of a stack of N decoders trained to decode along a secondary time axis that allows model- parameter updates based on N prediction steps. TeaForN can be used with a wide class of decoder architectures and requires minimal modifications from a standard teacher-forcing setup. Empirically, we show that TeaForN boosts generation quality on one Machine Translation benchmark, WMT 2014 English- French, and two News Summarization bench- marks, CNN/Dailymail and Gigaword. 1 Introduction Many state-of-the-art sequence generation mod- els are trained using a technique called teacher- forcing (Goodfellow et al., 2016). Teacher-forcing is popular because it improves sample efficiency and provides training stability, but models trained with teacher-forcing are known to suffer from is- sues such as exposure bias (Venkatraman et al., 2015; Bengio et al., 2015; Ding and Soricut, 2017) and a lack of differentiability across timesteps (i.e., training updates made when decoding at time-step t cannot fully propagate to time-step t−1). Previous attempts to address these issues include scheduled sampling (Bengio et al., 2015), parallel N-gram prediction (Yan et al., 2020), and sampling from previous predictions (Zhang et al., 2019). Our proposed method, Teacher-Forcing with N- grams (TeaForN), imposes few requirements on the decoder architecture and does not require curricu- lum learning or sampling model outputs. TeaForN fully embraces the teacher-forcing paradigm and extends it to N-grams, thereby addressing the prob- lem at the level of teacher-forcing itself. The advent of large-scale pretraining has pushed the state-of-the-art on Natural Language bench- marks to impressive heights, often showing gains across many tasks at once (Devlin et al., 2019; Raf- fel et al., 2019; Zhang et al., 2019). A negative consequence of this is the tendency towards large, data-hungry models, which have a negative impact on energy-consumption and accessibility (Strubell et al., 2019), as well as higher latency and produc- tion costs. As such, it is of increasing importance to develop techniques that counteract these tenden- cies. While TeaForN does increase training cost moderately, it can be used to drive down latency and inference cost, which dominate the overall cost of a production model. Many sequence generation models use beam search to improve generation quality (Vaswani et al., 2017; Raffel et al., 2019; Zhang et al., 2019; Yan et al., 2020). In contrast with greedy decoding, beam search keeps the k most-likely candidates at each decoding timestep. While beam search has proven to be a reliable technique for improv- ing output quality, previous work has shown that beam search actually degrades performance for …

Content truncated to 500 words