Centering-based Neural Coherence Modeling with Hierarchical Discourse Segments

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 7458–7472, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 7458 Centering-based Neural Coherence Modeling with Hierarchical Discourse Segments Sungho Jeon and Michael Strube Heidelberg Institute for Theoretical Studies gGmbH {sungho.jeon, michael.strube}@h-its.org Abstract Previous neural coherence models have fo- cused on identifying semantic relations be- tween adjacent sentences. However, they do not have the means to exploit structural infor- mation. In this work, we propose a coherence model which takes discourse structural infor- mation into account without relying on human annotations. We approximate a linguistic the- ory of coherence, Centering theory, which we use to track the changes of focus between dis- course segments. Our model first identifies the focus of each sentence, recognized with re- gards to the context, and constructs the struc- tural relationship for discourse segments by tracking the changes of the focus. The model then incorporates this structural information into a structure-aware transformer. We eval- uate our model on two tasks, automated es- say scoring and assessing writing quality. Our results demonstrate that our model, built on top of a pretrained language model, achieves state-of-the-art performance on both tasks. We next statistically examine the identified trees of texts assigned to different quality scores. Fi- nally, we investigate what our model learns in terms of theoretical claims1. 1 Introduction Coherence describes the semantic relation between elements of a text. It identifies a text passage as either a unified whole or a collection of unrelated sentences. The most well-known formal theory, Centering theory, determines the most salient item in each sentence, the center or focus, and tracks the changes of the focus (Grosz et al., 1995). Prior studies of coherence have mainly focused on mod- eling local coherence in Centering theory (Barzilay and Lapata, 2008). They aim to identify the seman- tic relations between adjacent sentences. However, 1Our code is available at: https://github.com/ sdeva14/emnlp20-centering-neural-hds coherence arises not only at the local level, but also at the document level giving insight into the structure of the discourse. Discourse structure represents the semantic or- ganization of a text. Incorporating structural in- formation into the model has been beneficial for diverse downstream tasks including text summa- rization (Marcu, 2000), translation (Guzm´an et al., 2014), sentiment analysis (Bhatia et al., 2015), and text classification (Ji and Smith, 2017). To identify discourse structure, earlier work adopts a supervised approach, relying on human an- notations (Hernault et al., 2010; Wang et al., 2017). However, annotating discourse structure is time consuming and costly. It requires annotators to understand not only the local context surrounding the target sentence but also higher level relations. Learning latent structure has been proposed to al- leviate this limitation. This approach induces the discourse structure from a text without annotations using an attention layer (Liu and Lapata, 2018). Recent work argues that, however, the learned trees have mostly little to no structure at the document level, and the model relies on specific linguistic cues (Ferracane et al., 2019). In …

Content truncated to 500 words