Learning Sentence Embeddings for Coherence Modelling and Beyond

Proceedings of Recent Advances in Natural Language Processing, pages 151–160, Varna, Bulgaria, Sep 2–4, 2019. https://doi.org/10.26615/978-954-452-056-4_018 151 Learning Sentence Embeddings for Coherence Modelling and Beyond Tanner Bohn Yining Hu Jinhang Zhang Charles X. Ling Department of Computer Science, Western University, London, ON, Canada {tbohn,yhu534,jzha337,charles.ling}@uwo.ca Abstract We present a novel and effective tech- nique for performing text coherence tasks while facilitating deeper insights into the data. Despite obtaining ever-increasing task performance, modern deep-learning approaches to NLP tasks often only pro- vide users with the final network deci- sion and no additional understanding of the data. In this work, we show that a new type of sentence embedding learned through self-supervision can be applied effectively to text coherence tasks while serving as a window through which deeper understanding of the data can be ob- tained. To produce these sentence em- beddings, we train a recurrent neural net- work to take individual sentences and pre- dict their location in a document in the form of a distribution over locations. We demonstrate that these embeddings, com- bined with simple visual heuristics, can be used to achieve performance competitive with state-of-the-art on multiple text co- herence tasks, outperforming more com- plex and specialized approaches. Addi- tionally, we demonstrate that these embed- dings can provide insights useful to writ- ers for improving writing quality and in- forming document structuring, and assist- ing readers in summarizing and locating information. 1 Introduction A goal of much of NLP research is to create tools that not only assist in completing tasks, but help gain insights into the text being analyzed. This is especially true of text coherence tasks, as users are likely to wonder where efforts should be focused How coherent is it? Suggest a coherent sentence order Algorithm: If dashed line is close to diagonal, high coherence. If far, low coherence. Algorithm: Take sentences in the order that the black dots appear along the x-axis. Result: Sentences 1 and 2 may be out of order, otherwise it is quite close, with a coherence of 0.73. Result: Suggested order: 2, 1, 4, 3, 5, 6. Figure 1: This paper abstract is analyzed by our sentence position model trained on academic ab- stracts. The sentence encodings (predicted posi- tion distributions) are shown below each sentence, where white is low probability and red is high. Po- sition quantiles are ordered from left to right. The first sentence, for example, is typical of the first sentence of abstracts as reflected in the high first- quantile value. For two text coherence tasks, we show the how the sentence encodings can easily be used to solve them. The black dots indicate the weighted average predicted position for each sen- tence. to improve writing or understand how text should be reorganized for improved coherence. By im- proving coherence, a text becomes easier to read and understand (Lapata and Barzilay, 2005), and in this work we particularly focus on measuring coherence in terms of sentence ordering. Many recent approaches to NLP tasks make use of end-to-end …

Content truncated to 500 words