Discourse Probing of Pretrained Language Models

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3849–3864 June 6–11, 2021. ©2021 Association for Computational Linguistics 3849 Discourse Probing of Pretrained Language Models Fajri Koto Jey Han Lau Timothy Baldwin School of Computing and Information Systems The University of Melbourne ffajri@student.unimelb.edu.au, jeyhan.lau@gmail.com, tbaldwin@unimelb.edu.au Abstract Existing work on probing of pretrained lan- guage models (LMs) has predominantly fo- cused on sentence-level syntactic tasks. In this paper, we introduce document-level dis- course probing to evaluate the ability of pre- trained LMs to capture document-level rela- tions. We experiment with 7 pretrained LMs, 4 languages, and 7 discourse probing tasks, and find BART to be overall the best model at capturing discourse — but only in its encoder, with BERT performing surprisingly well as the baseline model. Across the different mod- els, there are substantial differences in which layers best capture discourse information, and large disparities between models. 1 Introduction The remarkable development of pretrained lan- guage models (Devlin et al., 2019; Lewis et al., 2020; Lan et al., 2020) has raised questions about what precise aspects of language these models do and do not capture. Probing tasks offer a means to perform fine-grained analysis of the capabilities of such models, but most existing work has focused on sentence-level analysis such as syntax (Hewitt and Manning, 2019; Jawahar et al., 2019; de Vries et al., 2020), entities/relations (Papanikolaou et al., 2019), and ontological knowledge (Michael et al., 2020). Less is known about how well such models capture broader discourse in documents. Rhetorical Structure Theory is a framework for capturing how sentences are connected and describ- ing the overall structure of a document (Mann and Thompson, 1986). A number of studies have used pretrained models to classify discourse markers (Sileo et al., 2019) and discourse relations (Nie et al., 2019; Shi and Demberg, 2019), but few (Koto et al., to appear) have systematically investigated the ability of pretrained models to model discourse structure. Furthermore, existing work relating to discourse probing has typically focused exclusively Model Type #Param #Data Objective BERT Enc 110M 16GB MLM+NSP RoBERTa 110M 160GB MLM ALBERT 12M 16GB MLM+SOP ELECTRA 110M 16GB MLM+DISC GPT-2 Dec 117M 40GB LM BART Enc+Dec 121M 160GB DAE T5 110M 750GB DAE Table 1: Summary of all English pretrained language models used in this work. “MLM” = masked language model, “NSP” = next sentence prediction, “SOP” = sentence order prediction, “LM” = language model, “DISC” = discriminator, and “DAE” = denoising au- toencoder. on the BERT-base model, leaving open the ques- tion of how well these findings generalize to other models with different pretraining objectives, for different languages, and different model sizes. Our research question in this paper is: How much discourse structure do layers of different pretrained language models capture, and do the findings gen- eralize across languages? There are two contemporaneous related studies that have examined discourse modelling in pre- trained language models. Upadhye et al. (2020) analyzed how well two …

Content truncated to 500 words