Searching for fingerspelled content in American Sign Language

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 127 - 141 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Attention Temperature Matters in Abstractive Summarization Distillation Shengqiang Zhang1 ∗ †, Xingxing Zhang2∗, Hangbo Bao2†, Furu Wei2 1 Peking University 2 Microsoft Research Asia sq.zhang@pku.edu.cn {xizhang,t-habao,fuwei}@microsoft.com Abstract Recent progress of abstractive text summa- rization largely relies on large pre-trained sequence-to-sequence Transformer which are computationally expensive.models, This paper aims to distill these large models into smaller ones for faster inference and with minimal performance loss. Pseudo-labeling based methods are popular in sequence-to- sequence model distillation. In this paper, we find simply manipulating attention temper- atures in Transformers can make pseudo la- bels easier to learn for student models. Our experiments on three summarization datasets show our proposed method consistently im- proves vanilla pseudo-labeling based methods. Further empirical analysis shows that both pseudo labels and summaries produced by our students are shorter and more abstractive. Our code is available at https://github. com/Shengqiang-Zhang/plate. 1 Introduction Automatic document summarization is the task of rewriting a long document into its shorter form while still retaining its most important content. In the literature, there are mainly two kinds of meth- ods for summarization: extractive summarization and abstractive summarization (Nenkova and McK- eown, 2011). In this work, we focus on abstractive summarization, which is viewed as a sequence-to- sequence (Seq2Seq) learning problem, since re- cent abstractive models outperform their extrac- tive counterparts and can produce more concise summaries (Raffel et al., 2020; Lewis et al., 2020; Zhang et al., 2020; Liu and Lapata, 2019). Recent progress of abstractive summarization largely relies on large pre-trained Transformer models (Raffel et al., 2020; Lewis et al., 2020; Zhang et al., 2020; Liu and Lapata, 2019; Bao et al., 2020). With these ∗ Equal contribution. † Work done during the authors’ internships at Microsoft Research Asia. extremely large models, we can obtain state-of-the- art summarization results, but they are slow for online inference, which makes them difficult to be used in the production environment even with cutting-edge hardware. This paper aims to distill these large Transformer summarization models into smaller ones with minimal loss in performance. Knowledge distillation is a class of methods that leverage the output of a (large) teacher model to guide the training of a (small) student model. In classification tasks, it is typically done by minimiz- ing the distance between the teacher and student predictions (Hinton et al., 2015). As to Seq2Seq models, an effective distillation method is called pseudo-labeling (Kim and Rush, 2016), where the teacher model generates pseudo summaries for all documents in the training set and the resulting document–pseudo-summary pairs are used to train the student model. In this paper, we argue that attention distribu- tions of a Seq2Seq teacher model might be too sharp. As a result, pseudo labels generated from it are sub-optimal for student models. In the sum- marization task, we observe that 1) pseudo sum- maries generated from our teacher model copy …

Content truncated to 500 words