Neural RST-based Evaluation of Discourse Coherence

Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing, pages 664–671 December 4 - 7, 2020. c⃝2020 Association for Computational Linguistics 664 Neural RST-based Evaluation of Discourse Coherence Grigorii Guz∗1, Peyman Bateni∗1,2, Darius Muglich1, Giuseppe Carenini1 University of British Columbia1, Inverted AI2 {g.guz@cs, pbateni@cs, darius.muglich@alumni, carenini@cs}.ubc.ca Abstract This paper evaluates the utility of Rhetorical Structure Theory (RST) trees and relations in discourse coherence evaluation. We show that incorporating silver-standard RST features can increase accuracy when classifying coherence. We demonstrate this through our tree-recursive neural model, namely RST-Recursive, which takes advantage of the text’s RST features pro- duced by a state of the art RST parser. We eval- uate our approach on the Grammarly Corpus for Discourse Coherence (GCDC) and show that when ensembled with the current state of the art, we can achieve the new state of the art accuracy on this benchmark. Furthermore, when deployed alone, RST-Recursive achieves competitive accuracy while having 62% fewer parameters. 1 Introduction Discourse coherence has been the subject of much research in Computational Linguistics thanks to its widespread applications (Lai and Tetreault, 2018). Most current methods can be described as either stemming from explicit representations based on the Centering Theory (Grosz et al., 1994), or deep learning approaches that learn without the use of hand-crafted linguistic features. Our work explores a third research avenue based on the Rhetorical Structure Theory (RST) (Mann and Thompson, 1988). We hypothesize that texts of low/high coherence tend to adhere to different discourse structures. Thus, we pose that using even silver-standard RST features should help in sepa- rating coherent texts from incoherent ones. This stems from the definition of the coherence itself - as the writer of a document needs to follow spe- cific rules for building a clear narrative or argument structure in which the role of each constituent of the document should be appropriate with respect ∗Authors contributed equally people think he is smart because he did well in school LSTM LSTM E N S You know Mark, B N S LSTM Tree LSTM Tree LSTM ... R ... ... ... RST Tree RST-RecursiveNN Evidence Background Figure 1: Overview of RST-Recursive; EDU embed- dings are generated for the leaf nodes using the EDU network. Subsequently, the RST tree is recursively tra- versed bottom-up using the RST network. to its local and global context, and even existing discourse parsers should be able to predict a plausi- ble structure that is consistent across all coherent documents. However, if a parser has difficulty in- terpreting a given document, it will be more likely to produce unrealistic trees with improbable pat- terns of discourse relations between constituents. This idea was first explored by Feng et al. (2014), who followed an approach similar to Barzilay and Lapata (2008) by estimating entity transition like- lihoods, but instead using discourse relations (pre- dicted by a state of the art discourse parser (Feng and Hirst, 2014)) that entities participate …

Content truncated to 500 words