Contextual Dynamic Prompting for Response Generation in Task-oriented Dialog Systems

Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 3102–3111 May 2-6, 2023 ©2023 Association for Computational Linguistics Contextual Dynamic Prompting for Response Generation in Task-oriented Dialog Systems Sandesh Swamy AWS AI Labs sanswamy@amazon.com Narges Tabari AWS AI Labs nargesam@amazon.com Chacha Chen∗ University of Chicago chacha@uchicago.edu Rashmi Gangadharaiah AWS AI Labs rgangad@amazon.com Abstract Response generation is one of the critical com- ponents in task-oriented dialog systems. Exist- ing studies have shown that large pre-trained language models can be adapted to this task. The typical paradigm of adapting such ex- tremely large language models would be by fine-tuning on the downstream tasks which is not only time-consuming but also involves sig- nificant resources and access to fine-tuning data. Prompting (Schick and Schütze, 2020) has been an alternative to fine-tuning in many NLP tasks. In our work, we explore the idea of using prompting for response generation in task-oriented dialog systems. Specifically, we propose an approach that performs contex- tual dynamic prompting where the prompts are learnt from dialog contexts. We aim to dis- till useful prompting signals from the dialog context. On experiments with MultiWOZ 2.2 dataset (Zang et al., 2020), we show that contex- tual dynamic prompts improve response gener- ation in terms of combined score (Mehri et al., 2019a) by 3 absolute points, and a massive 20 points when dialog states are incorporated. Furthermore, human annotation on these con- versations found that agents which incorporate context were preferred over agents with vanilla prefix-tuning. 1 Introduction With the advent of large language models (LLMs), a vast majority of NLP tasks, including dialog sys- tems, further fine-tune these LMs for their down- stream tasks. Although these approaches pro- vide substantial improvements over traditional task- specific models (Ham et al., 2020; Hosseini-Asl et al., 2020; He et al., 2022), it is a time consum- ing process that also involves significant use of energy/resources in the form of compute. These ap- proaches also require tuning and storing parameters for each downstream task. ∗ Work done during an internship at AWS AI Labs A more recent line of work, explores “prompt- ing” LLMs to elicit the necessary knowledge re- quired for the downstream tasks (Shin et al., 2020; Gao et al., 2020; Schick and Schütze, 2020; Petroni et al., 2019; Lee et al., 2021; Zhu et al., 2022). Prompts composed of tokens or short pieces of text (discrete prompts) inserted at the end of the input examples. These prompts are typically man- ually defined based on the specific downstream task. The main motivation behind these approaches stems from the idea that the large corpora that these language models are trained on contain relevant in- formation which is pertinent to the task on hand. Adapter-tuning was proposed as an alternate ap- proach to fine-tuning. These methods only train task-specific layers that are inserted within pre- trained LMs. Such a lightweight approach that add about 4% task-specific parameters has shown to ob- tain comparable performances to their fine-tuning counterparts …

Content truncated to 500 words