Structural Pre-training for Dialogue Comprehension

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 5134–5145 August 1–6, 2021. ©2021 Association for Computational Linguistics 5134 Structural Pre-training for Dialogue Comprehension Zhuosheng Zhang1,2,3, Hai Zhao1,2,3,∗ 1Department of Computer Science and Engineering, Shanghai Jiao Tong University 2Key Laboratory of Shanghai Education Commission for Intelligent Interaction and Cognitive Engineering, Shanghai Jiao Tong University, Shanghai, China 3MoE Key Lab of Artificial Intelligence, AI Institute, Shanghai Jiao Tong University zhangzs@sjtu.edu.cn, zhaohai@cs.sjtu.edu.cn Abstract Pre-trained language models (PrLMs) have demonstrated superior performance due to their strong ability to learn universal lan- guage representations from self-supervised pre-training. However, even with the help of the powerful PrLMs, it is still challenging to ef- fectively capture task-related knowledge from dialogue texts which are enriched by corre- lations among speaker-aware utterances. In this work, we present SPIDER, Structural Pre- traIned DialoguE Reader, to capture dialogue exclusive features. To simulate the dialogue- like features, we propose two training objec- tives in addition to the original LM objectives: 1) utterance order restoration, which predicts the order of the permuted utterances in dia- logue context; 2) sentence backbone regular- ization, which regularizes the model to im- prove the factual correctness of summarized subject-verb-object triplets. Experimental re- sults on widely used dialogue benchmarks ver- ify the effectiveness of the newly introduced self-supervised tasks. 1 Introduction Recent advances in large-scale pre-training lan- guage models (PrLMs) have achieved remarkable successes in a variety of natural language pro- cessing (NLP) tasks (Peters et al., 2018; Radford et al., 2018; Devlin et al., 2019a; Yang et al., 2019; Clark et al., 2020; Zhang et al., 2020d). Provid- ing fine-grained contextualized embedding, these pre-trained models are widely employed as en- coders for various downstream NLP tasks. Al- though the PrLMs demonstrate superior perfor- ∗ Corresponding author. This paper was partially sup- ported by National Key Research and Development Pro- gram of China (No. 2017YFB0304100), Key Projects of National Natural Science Foundation of China (U1836222 and 61733011), Huawei-SJTU long term AI project, Cutting- edge Machine Reading Comprehension and Language Model. This work was supported by Huawei Noah’s Ark Lab. U1: how can i connect to URL streaming server ? U2: i think mplayer does it U3: i use kino , could be ? U4: kino is editing software . i don't think it supports media . U5: sorry EMOJI , i mean movie player U6: not sure about it . if it can't try vlc and mplayer too . U7: is mplayer aptable ? U8: yes but i 'm not sure if it is in the main repo . U9: have you ever updated your sources.list ? U10: i have no idea..i use adept on kubuntu . Figure 1: A multi-turn dialogue example. Different col- ors indicate the utterances from different speakers. mance due to their strong representation ability from self-supervised pre-training, it is still chal- lenging to effectively adapt task-related knowledge during the detailed task-specific training which is usually in a way …

Content truncated to 500 words