SentBS: Sentence-level Beam Search for Controllable Summarization

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 10256 - 10265 December 7-11, 2022 ©2022 Association for Computational Linguistics SentBS: Sentence-level Beam Search for Controllable Summarization Chenhui Shen ∗ 1,2 Liying Cheng ∗1,3 Lidong Bing†1 Yang You2 Luo Si1 1DAMO Academy, Alibaba Group 2 National University of Singapore 3Singapore University of Technology and Design {chenhui.shen, liying.cheng}@alibaba-inc.com {l.bing, luo.si}@alibaba-inc.com youy@comp.nus.edu.sg Abstract A wide range of control perspectives have been explored in controllable text generation. Structure-controlled summarization is recently proposed as a useful and interesting research di- rection. However, current structure-controlling methods have limited effectiveness in enforcing the desired structure. To address this limitation, we propose a sentence-level beam search gen- eration method (SentBS), where evaluation is conducted throughout the generation process to select suitable sentences for subsequent gen- erations. We experiment with different combi- nations of decoding methods to be used as sub- components by SentBS and evaluate results on the structure-controlled dataset MReD. Experi- ments show that all explored combinations for SentBS can improve the agreement between the generated text and the desired structure, with the best method significantly reducing the structural discrepancies suffered by the existing model, by approximately 68%. 1 1 Introduction Controllable text generation is receiving increas- ing attention due to its wide range of applications. Depending on the use cases, the controllable gen- eration tasks may focus on a wide range of control perspectives, such as entities (Narayan et al., 2022; Fan et al., 2018a), aspects (Hayashi et al., 2021), and keywords (Wang et al., 2021; He et al., 2020). Recently, Shen et al. (2022) propose a sentence- level labeled meta-review dataset, MReD, for the controllable summarization task from a new control perspective that focuses on controlling the struc- ture of the output summary. The input consists of several reviews on the same research paper, and a control sequence specifying the desired summary ∗Chenhui and Liying are under the Joint PhD Program between Alibaba and their corresponding universities. † Corresponding author. 1Our code and data will be available at https://github .com/Shen-Chenhui/SentBS. structure. For instance, with a control sequence of “abstract | strength | decision”, the generated output should be composed of a sentence that summarizes the contents of the paper, followed by a sentence discussing the strengths, then the last sentence giv- ing the final decision. Previous controllable summarization models are commonly fine-tuned on pre-trained transformer architectures (Vaswani et al., 2017) such as BART (Lewis et al., 2020) and Pegasus (Zhang et al., 2020a), with the control signals merged into the text input or prompts (Shen et al., 2022; Narayan et al., 2022; He et al., 2020; Keskar et al., 2019; Fan et al., 2018a). Previous works mainly focus on improving the summary’s similarity with the gold reference, leaving room for further improve- ment on the controllability. In particular, the best- performing model on the recently released MReD dataset still generates around 29% of the sentences that do not follow the control structure2, which is far from satisfactory. In this paper, …

Content truncated to 500 words