Alternating Recurrent Dialog Model with Large-scale Pre-trained Language Models

Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics, pages 1292–1301 April 19 - 23, 2021. ©2021 Association for Computational Linguistics 1292 Alternating Recurrent Dialog Model with Large-scale Pre-trained Language Models Qingyang Wu1, Yichi Zhang2, Yu Li1, Zhou Yu1 1University of California, Davis, 2Tsinghua University, {wilwu, yooli, joyu}@ucdavis.edu, zhangyic17@mails.tsinghua.edu.cn Abstract Existing dialog system models require exten- sive human annotations and are difficult to gen- eralize to different tasks. The recent success of large pre-trained language models has sug- gested the effectiveness of incorporating lan- guage priors in down-stream NLP tasks. How- ever, how much pre-trained language models can help dialog response generation is still un- der exploration. In this paper, we propose a simple, general, and effective framework: Al- ternating Recurrent Dialog Model (ARDM)1. ARDM models each speaker separately and takes advantage of large pre-trained language models. It requires no supervision from human annotations such as belief states or dialog acts to achieve effective conversations. ARDM out- performs or is on par with the state-of-the- art methods on two popular task-oriented di- alog datasets: CamRest676 and MultiWOZ. Moreover, we can generalize ARDM to more challenging, non-collaborative tasks such as persuasion. In the PersuasionForGood task, ARDM is capable of generating human-like re- sponses to persuade people to donate to a char- ity. 1 Introduction It has been a long-standing ambition for artificial intelligence researchers to create an intelligent con- versational agent that can generate human-like re- sponses. Recently, data-driven dialog models are more and more popular. However, most current state-of-the-art approaches still heavily rely on ex- tensive human annotations such as belief states and dialog acts (Lei et al., 2018). However, dialog con- tent can vary considerably in different dialog tasks. Having a different intent or dialog act annotation scheme for each task is costly and even impossible for tasks such as open-domain social chat. Thus, it 1https://github.com/qywu/ARDM is difficult to utilize these methods on challenging dialog tasks where dialog states and acts are diffi- cult to annotate such as persuasion and negotiation. Eric and Manning (2017) proposed a simple sequence-to-sequence architecture that requires no explicit annotations. The model learns to extract information from dialog history with attention and copy mechanism. However, due to the limited lan- guage modeling capability in the previous model, Sequicity (Lei et al., 2018), which uses belief states as inputs for supervision, outperforms Eric and Manning (2017)’s method significantly in the re- cent dialog datasets. But with the success of large pre-trained language models such as BERT (Devlin et al., 2019) and GPT-2 (Radford et al., 2019), we investigate how large-scale pre-trained language models can help dialog tasks. Previous sequence-to-sequence models are used to tackle documents with only one narrator. How- ever, in dialogs, two speakers have different roles; therefore, their language model distributions are very different from each other. To address this issue, we propose ARDM, a dialog model that encodes and decodes different speaker utterances in alternating order. This structure makes the model more flexible and efficient than …

Content truncated to 500 words