Long and Diverse Text Generation with Planning-based Hierarchical Variational Model

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 3257–3268, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 3257 Long and Diverse Text Generation with Planning-based Hierarchical Variational Model Zhihong Shao1, Minlie Huang1∗, Jiangtao Wen1, Wenfei Xu2, Xiaoyan Zhu1 1 Institute for Artificial Intelligence, State Key Lab of Intelligent Technology and Systems 1 Beijing National Research Center for Information Science and Technology 1 Department of Computer Science and Technology, Tsinghua University, Beijing 100084, China 2 Baozun, Shanghai, China szh19@mails.tsinghua.edu.cn, aihuang@tsinghua.edu.cn jtwen@tsinghua.edu.cn, xuwenfeilittle@gmail.com zxy-dcs@tsinghua.edu.cn Abstract Existing neural methods for data-to-text gen- eration are still struggling to produce long and diverse texts: they are insufficient to model input data dynamically during generation, to capture inter-sentence coherence, or to gener- ate diversified expressions. To address these issues, we propose a Planning-based Hierar- chical Variational Model (PHVM). Our model first plans a sequence of groups (each group is a subset of input items to be covered by a sentence) and then realizes each sentence con- ditioned on the planning result and the pre- viously generated context, thereby decompos- ing long text generation into dependent sen- tence generation sub-tasks. To capture expres- sion diversity, we devise a hierarchical latent structure where a global planning latent vari- able models the diversity of reasonable plan- ning and a sequence of local latent variables controls sentence realization. Experiments show that our model outperforms state-of-the- art baselines in long and diverse text genera- tion. 1 Introduction Data-to-text generation is to generate natural lan- guage texts from structured data (Gatt and Krah- mer, 2018), which has a wide range of applications (for weather forecast, game report, product de- scription, advertising document, etc.). Most neu- ral methods focus on devising encoding scheme and attention mechanism, namely, (1) exploiting input structure to learn better representation of in- put data (Lebret et al., 2016; Liu et al., 2018), and (2) devising attention mechanisms to better em- ploy input data (Mei et al., 2016; Liu et al., 2018; Nema et al., 2018) or to dynamically trace which part of input has been covered in generation (Kid- don et al., 2016). These models are able to pro- ∗*Corresponding author: Minlie Huang. duce fluent and coherent short texts in some appli- cations. However, to generate long and diverse texts such as product descriptions, existing methods are still unable to capture the complex semantic structures and diversified surface forms of long texts. First, existing methods are not good at modeling input data dynamically during genera- tion. Some neural methods (Kiddon et al., 2016; Feng et al., 2018) propose to record the accumu- lated attention devoted to each input item. How- ever, these records may accumulate errors in rep- resenting the state of already generated prefix, thus leading to wrong new attention weights. Second, inter-sentence coherence in long text generation is not well captured (Wiseman et al., 2017) due to the lack of high-level planning. Recent stud- ies propose to model …

Content truncated to 500 words