OPAL: Ontology-Aware Pretrained Language Model for End-to-End Task-Oriented Dialogue

OPAL: Ontology-Aware Pretrained Language Model for End-to-End Task-Oriented Dialogue Zhi Chen1, Yuncong Liu1, Lu Chen1∗, Su Zhu2, Mengyue Wu1, Kai Yu1∗ 1X-LANCE Lab, Department of Computer Science and Engineering MoE Key Lab of Artificial Intelligence, AI Institute, Shanghai Jiao Tong University State Key Lab of Media Convergence Production Technology and Systems, Beijing, China 2AISpeech Co., Ltd., Suzhou, China {zhenchi713, chenlusz, kai.yu}@sjtu.edu.cn Abstract This paper presents an ontology-aware pre- trained language model (OPAL) for end-to-end task-oriented dialogue (TOD). Unlike chit-chat dialogue models, task-oriented dialogue mod- els fulfill at least two task-specific modules: Dialogue state tracker (DST) and response generator (RG). The dialogue state consists of the domain-slot-value triples, which are regarded as the user’s constraints to search the domain-related databases. The large-scale task-oriented dialogue data with the annotated structured dialogue state usually are inacces- sible. It prevents the development of the pre- trained language model for the task-oriented dialogue. We propose a simple yet effective pretraining method to alleviate this problem, which consists of two pretraining phases. The first phase is to pretrain on large-scale con- textual text data, where the structured informa- tion of the text is extracted by the information extracting tool. To bridge the gap between the pretraining method and downstream tasks, we design two pretraining tasks: ontology- like triple recovery and next-text generation, which simulates the DST and RG, respectively. The second phase is to fine-tune the pretrained model on the TOD data. The experimental re- sults show that our proposed method achieves an exciting boost and obtains competitive performance even without any TOD data on CamRest676 and MultiWOZ benchmarks. 1 Introduction A task-oriented dialogue (TOD) system aims to assist users in accomplishing a specific task by interacting with natural language, for example, reserving a hotel or booking flight tickets. With the popularity of the industrial dialogue system, the ∗The corresponding authors are Lu Chen and Kai Yu. task-oriented dialogue system attracts extensive attention in research. The existing task-oriented dialogue system can be classified into two categories: pipeline format and end-to-end format. The pipeline TOD system (Ultes et al., 2017; Weisz et al., 2018) is composed of four modules: natural language understanding (NLU) (Quirk et al., 2015), dialogue state track- ing (DST) (Xu et al., 2020; Chen et al., 2020c), dialogue policy (DP) (Chen et al., 2018, 2019, 2020b), and natural language generation (NLG) (Wen et al., 2015; Li et al., 2016; Zhao et al., 2017). Since each module of the system is trained separately and executes sequentially, it faces two serious issues: error accumulation and high anno- tation cost. Thus, the end-to-end dialogue system (Lee et al., 2019b; Zhao et al., 2019) gradually becomes the research focus, which formulates the task-oriented dialogue as a sequence-to-sequence task. The dialogue state, database (DB) state, and the corresponding system response are directly concatenated together and flattened as a token se- quence. The DB state is the status of the domain- related database searched with the dialogue state, as shown in Figure 1. Thanks to the success of pretraining language …

Content truncated to 500 words