Towards Understanding Large-Scale Discourse Structures in Pre-Trained and Fine-Tuned Language Models

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2376 - 2394 July 10-15, 2022 ©2022 Association for Computational Linguistics Towards Understanding Large-Scale Discourse Structures in Pre-Trained and Fine-Tuned Language Models Patrick Huber and Giuseppe Carenini Department of Computer Science University of British Columbia Vancouver, BC, Canada, V6T 1Z4 {huberpat, carenini}@cs.ubc.ca Abstract With a growing number of BERTology works analyzing different components of pre-trained language models, we extend this line of re- search through an in-depth analysis of dis- course information in pre-trained and fine- tuned language models. We move beyond prior work along three dimensions: First, we de- scribe a novel approach to infer discourse struc- tures from arbitrarily long documents. Second, we propose a new type of analysis to explore where and how accurately intrinsic discourse is captured in the BERT and BART models. Finally, we assess how similar the generated structures are to a variety of baselines as well as their distributions within and between models. 1 Introduction Transformer-based machine learning models are an integral part of many recent improvements in Natural Language Processing (NLP). With their rise spearheaded by Vaswani et al. (2017), the pre-training/fine-tuning paradigm has gradually replaced previous approaches based on architec- ture engineering, with transformer models such as BERT (Devlin et al., 2019), BART (Lewis et al., 2020), RoBERTa (Liu et al., 2019) and others deliv- ering state-of-the-art performance on a wide variety of tasks. Besides their strong empirical results on most real-world problems, such as summarization (Zhang et al., 2020; Xiao et al., 2021a), question- answering (Joshi et al., 2020; O˘guz et al., 2021) and sentiment analysis (Adhikari et al., 2019; Yang et al., 2019), uncovering what kind of linguistic knowledge is captured by this new type of pre- trained language models (PLMs) has become a prominent question by itself. As part of this line of research, called BERTology (Rogers et al., 2020), researchers explore the amount of linguistic under- standing encapsulated in PLMs, exposed through either external probing tasks (Raganato and Tiede- mann, 2018; Zhu et al., 2020; Koto et al., 2021a) or unsupervised methods (Wu et al., 2020; Pandia et al., 2021). Previous work thereby either focuses on analyzing the syntactic structures (e.g., Hewitt and Manning (2019); Wu et al. (2020)), relations (Papanikolaou et al., 2019), ontologies (Michael et al., 2020) or, to a more limited extend, discourse related behaviour (Zhu et al., 2020; Koto et al., 2021a; Pandia et al., 2021). Generally speaking, while most previous BERTology works has focused on either sentence level phenomena or connections between adja- cent sentences, large-scale semantic and pragmatic structures (oftentimes represented as discourse trees or graphs) have been less explored. These structures (e.g., discourse trees) play a fundamen- tal role in expressing the intent of multi-sentential documents and, not surprisingly, have been shown to benefit many NLP tasks such as summarization (Gerani et al., 2019), sentiment analysis (Bhatia et al., 2015; Nejat et al., 2017; Hogenboom et al., 2015) and text …

Content truncated to 500 words