Dialogue-oriented Pre-training

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 2663–2673 August 1–6, 2021. ©2021 Association for Computational Linguistics 2663 Dialogue-oriented Pre-training Yi Xu1,2,3, Hai Zhao1,2,3,∗ 1 Department of Computer Science and Engineering, Shanghai Jiao Tong University 2 Key Laboratory of Shanghai Education Commission for Intelligent Interaction and Cognitive Engineering, Shanghai Jiao Tong University, Shanghai, China 3MoE Key Lab of Artificial Intelligence, AI Institute, Shanghai Jiao Tong University xuyi 2019@sjtu.edu.cn,zhaohai@cs.sjtu.edu.cn Abstract Pre-trained language models (PrLM) has been shown powerful in enhancing a broad range of downstream tasks including various dia- logue related ones. However, PrLMs are usu- ally trained on general plain text with com- mon language model (LM) training objectives, which cannot sufficiently capture dialogue ex- clusive features due to the limitation of such training setting, so that there is an immedi- ate need to fill the gap between a specific di- alogue task and the LM task. As it is unlikely to collect huge dialogue data for dialogue- oriented pre-training, in this paper, we pro- pose three strategies to simulate the conver- sation features on general plain text. Our proposed method differs from existing post- training methods that it may yield a general- purpose PrLM and does not individualize to any detailed task while keeping the capabil- ity of learning dialogue related features includ- ing speaker awareness, continuity and consis- tency. The resulted Dialog-PrLM is fine-tuned on three public multi-turn dialogue datasets and helps achieve significant and consistent improvement over the plain PrLMs. 1 Introduction Recently, pre-trained language models (PrLMs) have shown impressive improvements for various downstream NLP tasks (Zhou et al., 2020; Ouyang et al., 2021; Zhang and Zhao, 2021; Radford et al., 2018; Yang et al., 2019; Zhang et al., 2020c; Clark et al., 2020; Li et al., 2021), including the response selection task for multi-turn dialogues, which takes a dialogue history as input and aims to select a ∗ Corresponding author. This paper was partially sup- ported by National Key Research and Development Pro- gram of China (No. 2017YFB0304100), Key Projects of National Natural Science Foundation of China (U1836222 and 61733011), Huawei-SJTU long term AI project, Cutting- edge Machine Reading Comprehension and Language Model. This work was supported by Huawei Noah’s Ark Lab. Dialogue History: A: Could you please get me a train? B: Sure I can help you find a train. B: where are you coming from? B: What time do you need to leave by? A: I am leaving Leicester and I need to leave by 20:30. B: What is your destination and day of travel? A: Cambridge and on friday. A: Can I just get the travel time for the train? Thanks! Response: B: The first train leaving after 20:30 is 142 21:09 and the travel time is 105 minutes. Table 1: A multi-turn dialogue example with inter- leaved or continuous utterances between two speakers. most suitable response from a collection of an- swers (Zhou et al., 2016; Wu et al., 2017; Zhou et al., 2018b; Zhu et al., 2018; Zhang et …

Content truncated to 500 words