MPC-BERT: A Pre-Trained Language Model for Multi-Party Conversation Understanding

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 3682–3692 August 1–6, 2021. ©2021 Association for Computational Linguistics 3682 MPC-BERT: A Pre-Trained Language Model for Multi-Party Conversation Understanding Jia-Chen Gu1∗, Chongyang Tao2, Zhen-Hua Ling1, Can Xu2, Xiubo Geng2, Daxin Jiang2† 1National Engineering Laboratory for Speech and Language Information Processing, University of Science and Technology of China, Hefei, China 2Microsoft, Beijing, China gujc@mail.ustc.edu.cn, zhling@ustc.edu.cn, {chotao,caxu,xigeng,djiang}@microsoft.com Abstract Recently, various neural models for multi- party conversation (MPC) have achieved impressive improvements on a variety of tasks such as addressee recognition, speaker identification and response prediction. However, these existing methods on MPC usually represent interlocutors and utterances individually and ignore the inherent complicated structure in MPC which may provide crucial interlocutor and utterance semantics and would enhance the conversation understanding process. To this end, we present MPC-BERT, a pre-trained model for MPC understanding that considers learning who says what to whom in a unified model with several elaborated self-supervised tasks. Particularly, these tasks can be generally categorized into (1) interlocutor structure modeling including reply-to utterance recognition, identical speaker searching and pointer consistency distinction, and (2) utterance semantics modeling including masked shared utterance restoration and shared node detection. We evaluate MPC- BERT on three downstream tasks including addressee recognition, speaker identification and response selection. Experimental results show that MPC-BERT outperforms previous methods by large margins and achieves new state-of-the-art performance on all three downstream tasks at two benchmarks. 1 Introduction Building a conversational agent with intelligence has drawn significant attention from both academia and industry. Most of existing methods have studied understanding conversations between two participants, aiming to return an appropriate re- sponse either in a generation-based (Shang et al., ∗Work done during the internship at Microsoft. †Corresponding author. Speaker Utterance Addressee I.1 How can I setup if I want add new - server at xchat? I.2 From places, network servers, work I.1 group, his computer, and then I clicked on the shared folder. I.3 It did not allow you to see the files? I.2 I.2 It prompts for authentication and I I.3 don’t know what to put. I tried guest with no password. I.4 Put proper authentication in, then? I.2 I.3 I think you had kde on suse? I.2 Table 1: An MPC example in Ubuntu IRC channel. Here, “I.” is the abbreviation of “interlocutor”. 2015; Serban et al., 2016, 2017; Zhang et al., 2018b, 2020) or retrieval-based manner (Lowe et al., 2015; Wu et al., 2017; Zhou et al., 2018; Tao et al., 2019a,b; Gu et al., 2019a,b, 2020). Recently, researchers have paid more attention to a more practical and challenging scenario involving more than two participants, which is well known as multi- party conversation (MPC) (Ouchi and Tsuboi, 2016; Zhang et al., 2018a; Le et al., 2019; Hu et al., 2019). Table 1 shows an MPC example in the Ubuntu Internet Relay Chat (IRC) channel, which is composed of a sequence of (speaker, utterance, addressee) triples. In addition …

Content truncated to 500 words