Why Can’t Discourse Parsing Generalize? A Thorough Investigation of the Impact of Data Diversity

Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 3112–3130 May 2-6, 2023 ©2023 Association for Computational Linguistics Why Can’t Discourse Parsing Generalize? A Thorough Investigation of the Impact of Data Diversity Yang Janet Liu and Amir Zeldes Department of Linguistics Georgetown University {yl879, amir.zeldes}@georgetown.edu Abstract Recent advances in discourse parsing perfor- mance create the impression that, as in other NLP tasks, performance for high-resource lan- guages such as English is finally becoming re- liable. In this paper we demonstrate that this is not the case, and thoroughly investigate the impact of data diversity on RST parsing sta- bility. We show that state-of-the-art architec- tures trained on the standard English newswire benchmark do not generalize well, even within the news domain. Using the two largest RST corpora of English with text from multiple gen- res, we quantify the impact of genre diversity in training data for achieving generalization to text types unseen during training. Our re- sults show that a heterogeneous training regime is critical for stable and generalizable models, across parser architectures. We also provide error analyses of model outputs and out-of- domain performance. To our knowledge, this study is the first to fully evaluate cross-corpus RST parsing generalizability on complete trees, examine between-genre degradation within an RST corpus, and investigate the impact of genre diversity in training data composition. 1 Introduction Discourse parsing is the task of identifying and classifying the coherence relations that hold be- tween different parts of a text, such as recognizing that one sentence specifies the cause of events in another, or that a subordinate clause indicates the purpose of a main clause. In hierarchical frame- works, such as the Rhetorical Structure Theory (RST, Mann and Thompson 1988), parsers con- struct a labeled constituent tree of discourse units as shown in Figure 1. Such trees have numerous applications: retrieving specific relations (e.g. all CONCESSIONS made in any speech by some politi- cian, e.g. unit 26-27 in Figure 1), or hierarchically finding the most central unit in an arbitrary span of text (unit 24 in Figure 1). They can also con- tribute to downstream tasks such as text generation Figure 1: RST Fragment from GUM (Zeldes, 2017). (Maskharashvili et al., 2021) and summarization (e.g. Louis et al. 2010; Li et al. 2016; Xu et al. 2020; Hewett and Stede 2022) and to qualitative analysis and comparison of texts (Wan et al., 2019).1 Recent advances in NLP have resulted in increas- ingly accurate systems for tasks such as part-of- speech tagging (Heinzerling and Strube, 2019) and dependency parsing (Mrini et al., 2020), which are now commonly applied with confidence to novel data, at least for high-resource languages such as English. At the same time, rising scores on the main standard English RST benchmark, the RST Discourse Treebank (RST-DT, Carlson et al. 2003), with data from the 1989 Wall Street Journal (WSJ), create the impression that discourse parsing too is becoming reliable, and by proxy, applica- ble to arbitrary text. …

Content truncated to 500 words