Transformer-based Lexically Constrained Headline Generation

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 4085–4090 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 4085 Transformer-based Lexically Constrained Headline Generation Kosuke Yamada1∗ Yuta Hitomi2† Hideaki Tamori3 Ryohei Sasano1 Naoaki Okazaki4 Kentaro Inui5,6 Koichi Takeda1 1Nagoya University 2Insight Edge, Inc. 3The Asahi Shimbun Company 4Tokyo Institute of Technology 5Tohoku University 6RIKEN AIP yamada.kosuke@c.mbox.nagoya-u.ac.jp, yuta.hitomi@insightedge.jp, tamori-h@asahi.com, {sasano,takedasu}@i.nagoya-u.ac.jp, okazaki@c.titech.ac.jp, inui@ecei.tohoku.ac.jp Abstract This paper explores a variant of automatic headline generation methods, where a gener- ated headline is required to include a given phrase such as a company or a product name. Previous methods using Transformer-based models generate a headline including a given phrase by providing the encoder with addi- tional information corresponding to the given phrase. However, these methods cannot al- ways include the phrase in the generated headline. Inspired by previous RNN-based methods generating token sequences in back- ward and forward directions from the given phrase, we propose a simple Transformer- based method that guarantees to include the given phrase in the high-quality generated headline. We also consider a new headline generation strategy that takes advantage of the controllable generation order of Transformer. Our experiments with the Japanese News Cor- pus demonstrate that our methods, which are guaranteed to include the phrase in the gener- ated headline, achieve ROUGE scores compa- rable to previous Transformer-based methods. We also show that our generation strategy per- forms better than previous strategies. 1 Introduction Following the initial work of Rush et al. (2015), abstractive headline generation using the encoder- decoder model has been studied extensively (Chopra et al., 2016; Nallapati et al., 2016; Paulus et al., 2018). In the automatic headline generation for advertising articles, there are requests to include a given phrase such as a company or product name in the headline. Generating a headline that includes a given phrase has been considered one of the lexically con- strained sentence generation tasks. For these tasks, ∗Work done during an internship at The Asahi Shimbun Company †Work done at The Asahi Shimbun Company there are two major approaches. One approach is to select a plausible sentence including the given phrase from several candidate sentences generated from left to right (Hokamp and Liu, 2017; Ander- son et al., 2017; Post and Vilar, 2018). Although these methods can include multiple phrases in a generated sentence, they are computationally ex- pensive due to the large search space of the decod- ing process. In addition, since they try to force given phrases into sentences at every step of the generation process, these methods may harm the quality of the generated sentence (Liu et al., 2019). Another approach proposed by Mou et al. (2015) is to generate token sequences in backward and forward directions from the given phrase. Mou et al. (2016) proposed Sequence to Backward and Forward Sequences (Seq2BF), which applies the method of Mou et al. (2015) to the sequence-to- sequence (seq2seq) framework. They use an RNN- based model and adopt the best strategies proposed by Mou et al. (2015), …

Content truncated to 500 words