Select and Attend: Towards Controllable Content Selection in Text Generation

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 579–590, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 579 Select and Attend: Towards Controllable Content Selection in Text Generation Xiaoyu Shen1,2∗, Jun Suzuki3,4, Kentaro Inui3,4, Hui Su5 Dietrich Klakow1 and Satoshi Sekine4 1Spoken Language Systems (LSV), Saarland University, Germany 2Max Planck Institute for Informatics, Saarland Informatics Campus, Germany 3Tohoku University 4RIKEN AIP, Japan 5Pattern Recognition Center, Wechat AI, Tencent Inc, China Abstract Many text generation tasks naturally contain two steps: content selection and surface real- ization. Current neural encoder-decoder mod- els conflate both steps into a black-box archi- tecture. As a result, the content to be de- scribed in the text cannot be explicitly con- trolled. This paper tackles this problem by de- coupling content selection from the decoder. The decoupled content selection is human in- terpretable, whose value can be manually ma- nipulated to control the content of generated text. The model can be trained end-to-end without human annotations by maximizing a lower bound of the marginal likelihood. We further propose an effective way to trade-off between performance and controllability with a single adjustable hyperparameter. In both data-to-text and headline generation tasks, our model achieves promising results, paving the way for controllable content selection in text generation.1 1 Introduction Many text generation tasks, e.g., data-to-text, sum- marization and image captioning, can be naturally divided into two steps: content selection and sur- face realization. The generations are supposed to have two levels of diversity: (1) content-level di- versity reflecting multiple possibilities of content selection (what to say) and (2) surface-level di- versity reflecting the linguistic variations of ver- balizing the selected contents (how to say) (Re- iter and Dale, 2000; Nema et al., 2017). Recent neural network models handle these tasks with the encoder-decoder (Enc-Dec) framework (Sutskever et al., 2014; Bahdanau et al., 2015), which simul- taneously performs selecting and verbalizing in a ∗Work mostly done while at RIKEN AIP. Correspon- dence to xshen@mpi-inf.mpg.de 1The source code is available on https://github. com/chin-gyou/controllable-selection Source Sentence: The sri lankan government on Wednes- day announced the closure of government schools with im- mediate effect as a military campaign against tamil sepa- ratists escalated in the north of the country. Selected : sri lankan, closure, schools Text: sri lanka closes schools . Selected : sri lankan, Wednesday, closure, schools Text: sri lanka closes schools on Wednesday. Selected : sri lankan, closure, schools, military campaign Text: sri lanka shuts down schools amid war fears. Selected : sri lankan, announced, closure, schools Text: sri lanka declares closure of schools. Table 1: Headline generation examples from our model. We can generate text describing various con- tents by sampling different content selections. The se- lected source word and its corresponding realizations in the text are highlighted with the same color. black-box way. Therefore, both levels of diver- sity are entangled within the generation. This en- tanglement, however, sacrifices the controllability and …

Content truncated to 500 words