CoDEx: A Comprehensive Knowledge Graph Completion Benchmark

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 917–929, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 917 TOD-BERT: Pre-trained Natural Language Understanding for Task-Oriented Dialogue Chien-Sheng Wu, Steven Hoi, Richard Socher, and Caiming Xiong Salesforce Research [wu.jason, shoi, rsocher, cxiong]@salesforce.com Abstract The underlying difference of linguistic pat- terns between general text and task-oriented dialogue makes existing pre-trained language models less useful in practice. In this work, we unify nine human-human and multi-turn task-oriented dialogue datasets for language modeling. To better model dialogue behav- ior during pre-training, we incorporate user and system tokens into the masked language modeling. We propose a contrastive objec- tive function to simulate the response selec- tion task. Our pre-trained task-oriented dia- logue BERT (TOD-BERT) outperforms strong baselines like BERT on four downstream task- oriented dialogue applications, including in- tention recognition, dialogue state tracking, di- alogue act prediction, and response selection. We also show that TOD-BERT has a stronger few-shot ability that can mitigate the data scarcity problem for task-oriented dialogue. 1 Introduction Pre-trained models with self-attention encoder ar- chitectures (Devlin et al., 2018; Liu et al., 2019) have been commonly used in many NLP appli- cations. Such models are self-supervised based on a massive scale of general text corpora, such as English Wikipedia or books (Zhu et al., 2015). By further fine-tuning these representations, break- throughs have been continuously reported for vari- ous downstream tasks, especially natural language understanding. However, previous work (Rashkin et al., 2018; Wolf et al., 2019) shows that there are some defi- ciencies in the performance to apply fine-tuning on conversational corpora directly. One possible reason could be the intrinsic difference of linguistic patterns between human conversations and writing text, resulting in a large gap of data distributions (Bao et al., 2019). Therefore, pre-training dialogue language models using chit-chat corpora from so- cial media, such as Twitter or Reddit, has been recently investigated, especially for dialogue re- sponse generation (Zhang et al., 2019) and retrieval (Henderson et al., 2019b). Although these open- domain dialogues are diverse and easy-to-get, they are usually short, noisy, and without specific chat- ting goals. On the other hand, a task-oriented dialogue has explicit goals (e.g. restaurant reservation or ticket booking) and many conversational interac- tions. But each dataset is usually small and scat- tered because obtaining and labeling such data is time-consuming. Moreover, a task-oriented dia- logue has explicit user and system behaviors where a user has his/her goal, and a system has its be- lief and database information, which makes the language understanding component and dialogue policy learning more important than those chit-chat scenarios. This paper aims to prove this hypothesis: self- supervised language model pre-training using task- oriented corpora can learn better representations than existing pre-trained models for task-oriented downstream tasks. We emphasize that what we care about the most is not whether our pre-trained model can achieve state-of-the-art results on each down- stream task since most of the current best models are built on top of pre-trained models, …

Content truncated to 500 words