Learning Non-Autoregressive Models from Search for Unsupervised Sentence Summarization

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 7916 - 7929 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Learning Non-Autoregressive Models from Search for Unsupervised Sentence Summarization Puyuan Liu, Chenyang Huang, Lili Mou Dept. Computing Science, Alberta Machine Intelligence Institute (Amii) University of Alberta, Canada {puyuan, chuang8}@ualberta.ca, doublepower.mou@gmail.com Abstract Text summarization aims to generate a short summary for an input text. In this work, we propose a Non-Autoregressive Unsupervised Summarization (NAUS) approach, which does not require parallel data for training. Our NAUS first performs edit-based search to- wards a heuristically defined score, and gener- ates a summary as pseudo-groundtruth. Then, we train an encoder-only non-autoregressive Transformer based on the search result. We also propose a dynamic programming ap- proach for length-control decoding, which is important for the summarization task. Ex- periments on two datasets show that NAUS achieves state-of-the-art performance for unsu- pervised summarization, yet largely improving inference efficiency. Further, our algorithm is able to perform explicit length-transfer sum- mary generation.1 1 Introduction Text summarization is an important natural lan- guage processing (NLP) task, aiming at generating concise summaries for given texts while preserving the key information. It has extensive real-world applications such as headline generation (Nenkova et al., 2011). In this paper, we focus on the set- ting of sentence summarization (Rush et al., 2015; Filippova et al., 2015). State-of-the-art text summarization models are typically trained in a supervised way with large training corpora, comprising pairs of long texts and their summaries (Zhang et al., 2020; Aghajanyan et al., 2020, 2021). However, such parallel data are expensive to obtain, preventing the applications to less popular domains and less spoken languages. Unsupervised text generation has been attracting increasing interest, because it does not require par- allel data for training. One widely used approach 1Our code, model, and output are released at: https: //github.com/MANGA-UOFA/NAUS is to compress a long text into a short one, and to reconstruct it to the long text by a cycle consis- tency loss (Miao and Blunsom, 2016; Wang and Lee, 2018; Baziotis et al., 2019). Due to the in- differentiability of the compressed sentence space, such an approach requires reinforcement learning (or its variants), which makes the training difficult (Kreutzer et al., 2021). Recently, Schumann et al. (2020) propose an edit-based approach for unsupervised summariza- tion. Their model maximizes a heuristically defined scoring function that evaluates the quality (fluency and semantics) of the generated summary, achiev- ing higher performance than cycle-consistency methods. However, the search approach is slow in inference because hundreds of search steps are needed for each data sample. Moreover, their ap- proach can only select words from the input sen- tence with the word order preserved. Thus, it is restricted and may generate noisy summaries due to the local optimality of search algorithms. To address the above drawbacks, we propose a Non-Autoregressive approach to Unsupervised Summarization (NAUS). The idea is to perform search as in Schumann et al. (2020) and, inspired by Li …

Content truncated to 500 words