Linguistic Versus Latent Relations for Modeling Coherent Flow in Paragraphs

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 5809–5815, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 5809 Linguistic Versus Latent Relations for Modeling Coherent Flow in Paragraphs Dongyeop Kang Hiroaki Hayashi Alan W Black Eduard Hovy Carnegie Mellon University, Pittsburgh, PA, USA {dongyeok,hiroakih,awb,hovy}@cs.cmu.edu Abstract Generating a long, coherent text such as a paragraph requires a high-level control of dif- ferent levels of relations between sentences (e.g., tense, coreference). We call such a log- ical connection between sentences as a (para- graph) flow. In order to produce a coherent flow of text, we explore two forms of inter- sentential relations in a paragraph: one is a human-created linguistical relation that forms a structure (e.g., discourse tree) and the other is a relation from latent representation learned from the sentences themselves. Our two pro- posed models incorporate each form of re- lations into document-level language models: the former is a supervised model that jointly learns a language model as well as discourse relation prediction, and the latter is an unsu- pervised model that is hierarchically condi- tioned by a recurrent neural network (RNN) over the latent information. Our proposed models with both forms of relations outper- form the baselines in partially conditioned paragraph generation task. Our codes and data are publicly available1. 1 Introduction When composing multiple sentences into a para- graph, as in novels or academic papers, we often make design decisions in advance (Byrne, 1979) such as topic introduction and content ordering to ensure better coherence of the text. For instance, McKeown (1985); Swan (2002) proposed effec- tive patterns for scientific writing: a hypothesis at first, followed by supporting sentences to val- idate the hypothesis, and lastly a concluding sen- tence. We call such a logical connection between sentences in a written paragraph as a flow. A co- herent flow between sentences requires an under- standing of various factors including tense, coref- erence, plans (Appelt, 1982; Hovy, 1991), scripts 1https://github.com/dykang/flownet (Tomkins, 1978) and several others. We focus on the paragraph-level plan between sentences. In text planning, underlying relations in text are broadly categorized into two forms: an explicit human-defined relation (e.g., a discourse tree) (Reiter and Dale, 2000) or an implicitly learned la- tent relation (Yang et al., 2016). While the former is defined and manuallly annotated based on lin- guistic theories, the latter is simply determinable from how people in fact put sentences together. In this work, we provide an empirical comparison be- tween a linguistically-informed and a latent form of relations in context of a paragraph generation. We compare the effectiveness of the two forms of relations using language modeling for para- graph generation. Due to the different charac- teristics of the two forms, we employ compara- ble but different components in addition to the base language model. For linguistic relations (e.g., discourse), we cast the problem into multi-task learning of supervised language modeling and dis- course relation prediction. …

Content truncated to 500 words