Discourse Structure Extraction from Pre-Trained and Fine-Tuned Language Models in Dialogues

Findings of the Association for Computational Linguistics: EACL 2023, pages 2562–2579 May 2-6, 2023 ©2023 Association for Computational Linguistics Discourse Structure Extraction from Pre-Trained and Fine-Tuned Language Models in Dialogues Chuyuan Li1, Patrick Huber2, Wen Xiao2 Maxime Amblard1, Chloé Braud3, Giuseppe Carenini2 1 Université de Lorraine, CNRS, Inria, LORIA, F-54000 Nancy, France 2 University of British Columbia, V6T 1Z4, Vancouver, BC, Canada 3 IRIT, Université de Toulouse, CNRS, ANITI, Toulouse, France 1{firstname.name}@loria.fr, 3chloe.braud@irit.fr, 2{huberpat, xiaowen3, carenini}@cs.ubc.ca Abstract Discourse processing suffers from data sparsity, especially for dialogues. As a result, we ex- plore approaches to build discourse structures for dialogues, based on attention matrices from Pre-trained Language Models (PLMs). We investigate multiple tasks for fine-tuning and show that the dialogue-tailored Sentence Order- ing task performs best. To locate and exploit discourse information in PLMs, we propose an unsupervised and a semi-supervised method. Our proposals thereby achieve encouraging re- sults on the STAC corpus, with F1 scores of 57.2 and 59.3 for the unsupervised and semi- supervised methods, respectively. When re- stricted to projective trees, our scores improved to 63.3 and 68.1. 1 Introduction In recent years, the availability of accurate tran- scription methods and the increase in online com- munication have led to a vast rise in dialogue data, necessitating the development of automatic analy- sis systems. For example, summarization of meet- ings or exchanges with customer service agents could be used to enhance collaborations or analyze customers issues (Li et al., 2019; Feng et al., 2021); machine reading comprehension in the form of question-answering could improve dialogue agents’ performance and help knowledge graph construc- tion (He et al., 2021; Li et al., 2021). However, simple surface-level features are oftentimes not sufficient to extract valuable information from con- versations (Qin et al., 2017). Rather, we need to understand the semantic and pragmatic relation- ships organizing the dialogue, for example through the use of discourse information. Along this line, several discourse frameworks have been proposed, underlying a variety of anno- tation projects. For dialogues, data has been pri- marily annotated within the Segmented Discourse Representation Theory (SDRT) (Asher et al., 2003). Figure 1: Excerpt of dependency structures in file s2- leagueM-game4, STAC. Red links are non-projective. Discourse structures are thereby represented as de- pendency graphs with arcs linking spans of text and labeled with semantico-pragmatic relations (e.g. Acknowledgment (Ack) or Question-Answer Pair (QAP)). Figure 1 shows an example from the Strate- gic Conversations corpus (STAC) (Asher et al., 2016). Discourse processing refers to the retrieval of the inherit structure of coherent text, and is of- ten separated into three tasks: EDU segmentation, structure building (or attachment), and relation pre- diction. In this work, we focus on the automatic extraction of (naked) structures without discourse relations. This serves as a first critical step in creat- ing a full discourse parser. It is important to note that naked structures have already been shown to be valuable features for specific tasks. Louis et al. (2010) mentioned that they are the most reliable in- dicator of …

Content truncated to 500 words