Pre-training Multi-party Dialogue Models with Latent Discourse Inference

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 9584–9599 July 9-14, 2023 ©2023 Association for Computational Linguistics Pre-training Multi-party Dialogue Models with Latent Discourse Inference Yiyang Li1,2,∗, Xinting Huang3,†, Wei Bi3 and Hai Zhao1,2,† 1 Department of Computer Science and Engineering, Shanghai Jiao Tong University 2 Key Laboratory of Shanghai Education Commission for Intelligent Interaction and Cognitive Engineering, Shanghai Jiao Tong University 3 NLP Center, Tencent AI-Lab eric-lee@sjtu.edu.cn, zhaohai@cs.sjtu.edu.cn {timxinhuang,victoriabi}@tencent.com Abstract Multi-party dialogues are more difficult for models to understand than one-to-one two- party dialogues, since they involve multiple interlocutors, resulting in interweaving reply-to relations and information flows. To step over these obstacles, an effective way is to pre-train a model that understands the discourse struc- ture of multi-party dialogues, namely, to whom each utterance is replying. However, due to the lack of explicitly annotated discourse labels in multi-party dialogue corpora, previous works fail to scale up the pre-training process by putting aside the unlabeled multi-party conver- sational data for nothing. To fully utilize the un- labeled data, we propose to treat the discourse structures as latent variables, then jointly infer them and pre-train the discourse-aware model by unsupervised latent variable inference meth- ods. Experiments on multiple downstream tasks show that our pre-trained model outper- forms strong baselines by large margins and achieves state-of-the-art (SOTA) results, justi- fying the effectiveness of our method. The offi- cial implementation of this paper is available at https://github.com/EricLee8/MPD_EMVI. 1 Introduction Dialogue system is an important area that has been studied for a long time in natural language process- ing field. Different from plain texts, dialogues are harder for models to understand since they are full of informal, colloquial expressions, and many el- lipses (Yang and Choi, 2019; Reddy et al., 2019; Li et al., 2022). Among them, multi-party dialogues are even more complex since they involve multi- ple interlocutors, resulting in interweaving reply-to relations and information flows (Gu et al., 2021; Sun et al., 2021; Gu et al., 2022b). Specifically, in multi-party dialogues, the current utterance can be ∗Work done while interning at Tencent AI Lab. † Corresponding author. This paper was partially sup- ported by Key Projects of National Natural Science Founda- tion of China (U1836222 and 61733011). a reply to any preceding utterance in the dialogue history, forming complex discourse structures. Intuitively, it is important for models to perceive the discourse structures, or in other words, to whom each utterance is replying, when comprehending multi-party dialogues. This intuition is in line with the process we humans participate in multi-party dialogues: we first read or listen to the dialogue history, knowing who speaks what to whom, then choose an utterance as the addressee, and finally utter a response. Literature has also justified that incorporating the discourse knowledge into models is beneficial for better understanding multi-party dialogues (Li et al., 2020; Jia et al., 2020; Li and Zhao, 2021; Ma et al., 2022). Unfortunately, the process of choosing addressees is a naturally un- observable action, …

Content truncated to 500 words