On Sparsifying Encoder Outputs in Sequence-to-Sequence Models

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 2888–2900 August 1–6, 2021. ©2021 Association for Computational Linguistics 2888 On Sparsifying Encoder Outputs in Sequence-to-Sequence Models Biao Zhang1 Ivan Titov1,2 Rico Sennrich3,1 1School of Informatics, University of Edinburgh 2ILLC, University of Amsterdam 3Department of Computational Linguistics, University of Zurich B.Zhang@ed.ac.uk, ititov@inf.ed.ac.uk, sennrich@cl.uzh.ch Abstract Sequence-to-sequence models usually transfer all encoder outputs to the decoder for genera- tion. In this work, by contrast, we hypothesize that these encoder outputs can be compressed to shorten the sequence delivered for decoding. We take Transformer as the testbed and intro- duce a layer of stochastic gates in-between the encoder and the decoder. The gates are regular- ized using the expected value of the sparsity- inducing L0 penalty, resulting in completely masking-out a subset of encoder outputs. In other words, via joint training, the L0DROP layer forces Transformer to route information through a subset of its encoder states. We in- vestigate the effects of this sparsification on two machine translation and two summariza- tion tasks. Experiments show that, depending on the task, around 40–70% of source encod- ings can be pruned without significantly com- promising quality. The decrease of the output length endows L0DROP with the potential of improving decoding efficiency, where it yields a speedup of up to 1.65× on document sum- marization and 1.20× on character-based ma- chine translation against the standard Trans- former. We analyze the L0DROP behaviour and observe that it exhibits systematic pref- erences for pruning certain word types, e.g., function words and punctuation get pruned most. Inspired by these observations, we ex- plore the feasibility of specifying rule-based patterns that mask out encoder outputs based on information such as part-of-speech tags, word frequency and word position. 1 1 Introduction Neural sequence-to-sequence (Seq2Seq) models have dominated various text generation tasks, including machine translation (Vaswani et al., 1Source code is available at https://github.com/ bzhangGo/zero. 0.0 0.5 1.0 1.5 2.0 2.5 3.0 Summed Attention Weight per Source Word 0 500 1000 1500 2000 Frequency Figure 1: Distribution of the summed attention weight per source word estimated on the English-German WMT14 test set. For each (source sentence, translation) pair, we extract the attention matrices from all encoder-decoder attention sublay- ers in Transformer and average them over different (8) heads and (6) layers. The attention value for each source word is summed over all target words in the translation. Higher atten- tion weights suggest larger impacts on translation. Around 49.7% source words get attention weights of less than 0.6, compared to the mean value of 1.03. 2017) and abstractive document summariza- tion (Gehrmann et al., 2018; Liu and Lapata, 2019). These models generally follow the encoder-decoder paradigm, where the encoder interprets source con- text and converts source words into vector represen- tations such that the decoder has sufficient informa- tion to predict the target sequence. Early Seq2Seq models (Sutskever et al., 2014; Cho et al., 2014) provided only the last and/or first encoder states to the decoder. In contrast, modern approaches rely on the …

Content truncated to 500 words