DS-TOD: Efficient Domain Specialization for Task-Oriented Dialog

Findings of the Association for Computational Linguistics: ACL 2022, pages 891 - 904 May 22-27, 2022 c⃝2022 Association for Computational Linguistics DS-TOD: Efficient Domain Specialization for Task-Oriented Dialog Chia-Chien Hung1, Anne Lauscher2, Simone Paolo Ponzetto1, and Goran Glavaš1,3 1Data and Web Science Group, University of Mannheim, Germany 2MilaNLP, Bocconi University, Italy 3Center for Information and Language Processing, LMU Munich, Germany {chia-chien, simone, goran}@informatik.uni-mannheim.de anne.lauscher@unibocconi.it Abstract Recent work has shown that self-supervised dialog-specific pretraining on large conversa- tional datasets yields substantial gains over tra- ditional language modeling (LM) pretraining in downstream task-oriented dialog (TOD). These approaches, however, exploit general dialogic corpora (e.g., Reddit) and thus presumably fail to reliably embed domain-specific knowl- edge useful for concrete downstream TOD do- mains. In this work, we investigate the ef- fects of domain specialization of pretrained lan- guage models (PLMs) for TOD. Within our DS-TOD framework, we first automatically ex- tract salient domain-specific terms, and then use them to construct DOMAINCC and DO- MAINREDDIT – resources that we leverage for domain-specific pretraining, based on (i) masked language modeling (MLM) and (ii) re- sponse selection (RS) objectives, respectively. We further propose a resource-efficient and modular domain specialization by means of do- main adapters – additional parameter-light lay- ers in which we encode the domain knowledge. Our experiments with prominent TOD tasks – dialog state tracking (DST) and response re- trieval (RR) – encompassing five domains from the MULTIWOZ benchmark demonstrate the effectiveness of DS-TOD. Moreover, we show that the light-weight adapter-based specializa- tion (1) performs comparably to full fine-tuning in single domain setups and (2) is particularly suitable for multi-domain specialization, where besides advantageous computational footprint, it can offer better TOD performance. 1 Introduction Task-oriented dialog (TOD), where conversational agents help users complete concrete tasks (e.g., book flights or order food), has arguably been one of the most prominent NLP applications in recent years, both in academia (Budzianowski et al., 2018; Henderson et al., 2019c; Liu et al., 2021a, inter alia) and industry (e.g., Yan et al., 2017; Hender- son et al., 2019b). Like for most other NLP tasks, fine-tuning of pretrained language models (PLMs) like BERT (Devlin et al., 2019) and GPT-2 (Rad- ford et al., 2019) pushed the state-of-the-art in TOD tasks (Budzianowski and Vuli´c, 2019; Hosseini-Asl et al., 2020), with LM pretraining at the same time alleviating the need for large labeled datasets (Ra- madan et al., 2018). More recent TOD work recognized the idiosyn- crasy of dialog – i.e., that dialogs represent inter- leaved exchanges of utterances between two (or more) participants – and proposed pretraining ob- jectives specifically tailored for dialogic corpora (Henderson et al., 2019c; Wu et al., 2020; Bao et al., 2020, inter alia). For instance, Wu et al. (2020) pre- train their TOD-BERT model on the concatenation of nine human-to-human multi-turn dialog datasets. Similarly, Henderson et al. (2019c, 2020) pretrain a general-purpose dialog encoder on a large corpus from Reddit by means of response selection objec- tives. Encoding dialogic linguistic knowledge in this way led to significant …

Content truncated to 500 words