The Change that Matters in Discourse Parsing: Estimating the Impact of Domain Shift on Parser Error

Findings of the Association for Computational Linguistics: ACL 2022, pages 824 - 845 May 22-27, 2022 c⃝2022 Association for Computational Linguistics The Change that Matters in Discourse Parsing: Estimating the Impact of Domain Shift on Parser Error Katherine Atwell†⋆, Anthony Sicilia‡⋆, Seong Jae Hwang§, Malihe Alikhani†‡ †Department of Computer Science and ‡Intelligent Systems Program, University of Pittsburgh §Department of Artificial Intelligence, Yonsei University {kaa139, anthonysicilia}@pitt.edu, seongjae@yonsei.ac.kr, malihe@pitt.edu Abstract Discourse analysis allows us to attain infer- ences of a text document that extend beyond the sentence-level. The current performance of discourse models is very low on texts outside of the training distribution’s coverage, dimin- ishing the practical utility of existing models. There is need for a measure that can inform us to what extent our model generalizes from the training to the test sample when these sam- ples may be drawn from distinct distributions. While this can be estimated via distribution shift, we argue that this does not directly cor- relate with change in the observed error of a classifier (i.e. error-gap). Thus, we propose to use a statistic from the theoretical domain adap- tation literature which can be directly tied to error-gap. We study the bias of this statistic as an estimator of error-gap both theoretically and through a large-scale empirical study of over 2400 experiments on 6 discourse datasets from domains including, but not limited to: news, biomedical texts, TED talks, Reddit posts, and fiction. Our results not only motivate our pro- posal and help us to understand its limitations, but also provide insight on the properties of dis- course models and datasets which improve per- formance in domain adaptation. For instance, we find that non-news datasets are slightly eas- ier to transfer to than news datasets when the training and test sets are very different. Our code and an associated Python package are available to allow practitioners to make more informed model and dataset choices.1 1 Introduction Coherence analysis of text is a key area of natural language processing. Discourse parsing models are trained on a dataset annotated according to a discourse framework, wherein the discourse struc- ture and how the discourse units are connected ⋆K. Atwell and A. Sicilia contributed equally. §Work done while at University of Pittsburgh. 1https://github.com/anthonysicilia/change-that-matters- ACL2022 Figure 1: Solid/hollow shapes indicate training/test set, while circles/squares indicate the correct labels. (A) Vertical shift is easily identified, but the classifier (dot- ted line) does well on both domains. (B) In the feature space, shift is imperceptible, but the classifier assigns the incorrect relation label to each point in the test set. In both, identifiable shift does not correlate with the classifier’s ability to correctly predict the discourse rela- tion are identified and labeled. Some discourse frame- works (Miltsakaki et al., 2004; Prasad et al., 2008; Webber et al., 2019) focus on shallow relations be- tween two individual discourse units, while others (Carlson et al., 2001; Lascarides and Asher, 2008) focus on learning a more hierarchical structure. Discourse models have been shown to improve performance in …

Content truncated to 500 words