Topic-Guided Coherence Modeling for Sentence Ordering by Preserving Global and Local Information

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 2273–2283, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 2273 Topic-Guided Coherence Modeling for Sentence Ordering by Preserving Global and Local Information Byungkook Oh, Seungmin Seo, Cheolheon Shin, Eunju Jo, Kyong-Ho Lee Department of Computer Science, Yonsei University Seoul, South Korea {bkoh, smseo, chshin, ejjo}@icl.yonsei.ac.kr khlee89@yonsei.ac.kr Abstract We propose a novel topic-guided coherence modeling (TGCM) for sentence ordering. Our attention based pointer decoder directly utilize sentence vectors in a permutation-invariant manner, without being compressed into a sin- gle fixed-length vector as the paragraph repre- sentation. Thus, TGCM can improve global dependencies among sentences and preserve relatively informative paragraph-level seman- tics. Moreover, to predict the next sentence, we capture topic-enhanced sentence-pair in- teractions between the current predicted sen- tence and each next-sentence candidate. With the coherent topical context matching, we promote local dependencies that help iden- tify the tight semantic connections for sen- tence ordering. The experimental results show that TGCM outperforms state-of-the-art mod- els from various perspectives. 1 Introduction Modeling the coherence among sentences to com- pute their gold order is one of the fundamen- tal tasks in Natural Language Processing (NLP) with many applications such as document model- ing (Narayan et al., 2018a), extractive document summarization (Jadhav and Rajan, 2018; Nallapati et al., 2017), question answering (Yu et al., 2018; Liu et al., 2017), conversational analysis (Zeng et al., 2018), automated text generation (Guo et al., 2018), and image captioning (Anderson et al., 2018). The coherence helps readers to improve reading comprehension and better understand the intent of a document. Sentence ordering is a set- to-sequence problem, which aims to identify the correct order of a sentence set. To do this, various studies on sentence ordering typically combine the coherent features extracted from sentences. In recent years, most of the traditional ap- proaches to sentence ordering are designed based on a pairwise strategy (Chen et al., 2016; Agrawal et al., 2016; Li and Jurafsky, 2016). The sen- tence pair ordering (SPO) models determine the relative order within a sentence pair via neural networks based semantic matching, which com- putes the relevance between the two sentence vec- tors. However, such models cause combinatorial optimization problems because search algorithms (e.g., beam search) are necessary to find the most optimal permutation. Since the SPO models only focus on the sentence-pair interactions (i.e., local dependency), they have trouble in capturing the interactions among three or more input sentences (i.e., global dependency) in an entire paragraph. More recently, state-of-the-art models aim to put randomly sorted sentences into a coherent paragraph with the correct order so that the whole sentences have the highest coherence probabil- ity (Vinyals et al., 2015a; Gong et al., 2016; Lo- geswaran et al., 2018; Cui et al., 2018). Unlike the SPO models, these models can perform the sen- tence set ordering (SSO) task for an entire para- graph based on the pointer network (Vinyals …

Content truncated to 500 words