Example-Driven Intent Prediction with Observers

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2979–2992 June 6–11, 2021. ©2021 Association for Computational Linguistics 2979 Example-Driven Intent Prediction with Observers Shikib Mehri♠∗ Mihail Eric♦ ♠Language Technologies Institute, Carnegie Mellon University, ♦Amazon Alexa AI amehri@cs.cmu.edu, mihaeric@amazon.com Abstract A key challenge of dialog systems research is to effectively and efficiently adapt to new domains. A scalable paradigm for adapta- tion necessitates the development of general- izable models that perform well in few-shot settings. In this paper, we focus on the intent classification problem which aims to identify user intents given utterances addressed to the dialog system. We propose two approaches for improving the generalizability of utter- ance classification models: (1) observers and (2) example-driven training. Prior work has shown that BERT-like models tend to attribute a significant amount of attention to the [CLS] token, which we hypothesize results in diluted representations. Observers are tokens that are not attended to, and are an alternative to the [CLS] token as a semantic representation of utterances. Example-driven training learns to classify utterances by comparing to examples, thereby using the underlying encoder as a sen- tence similarity model. These methods are complementary; improving the representation through observers allows the example-driven model to better measure sentence similari- ties. When combined, the proposed methods attain state-of-the-art results on three intent prediction datasets (BANKING77, CLINC150, HWU64) in both the full data and few-shot (10 examples per intent) settings. Furthermore, we demonstrate that the proposed approach can transfer to new intents and across datasets without any additional training. 1 Introduction Task-oriented dialog systems aim to satisfy a user goal in the context of a specific task such as book- ing flights (Hemphill et al., 1990), providing transit information (Raux et al., 2005), or acting as a tour guide (Budzianowski et al., 2018). Task-oriented dialog systems must first understand the user’s goal ∗Work done while Shikib was at Amazon by extracting meaning from a natural language ut- terance. This problem is known as intent predic- tion and is a vital component of task-oriented dia- log systems (Hemphill et al., 1990; Coucke et al., 2018). Given the vast space of potential domains, a key challenge of dialog systems research is to effectively and efficiently adapt to new domains (Rastogi et al., 2019). Rather than adapting to new domains by relying on large amounts of domain- specific data, a scalable paradigm for adaptation ne- cessitates the development of generalizable models that perform well in few-shot settings (Casanueva et al., 2020; Mehri et al., 2020). The task of intent prediction can be character- ized as a two step process: (1) representation (mapping a natural language utterance to a seman- tically meaningful representation) and (2) predic- tion (inferring an intent given a latent represen- tation). These two steps are complementary and interdependent, thereby necessitating that they be jointly improved. Therefore, to enhance the domain adaptation abilities of intent classification systems we propose to (1) improve the representation step through observers …

Content truncated to 500 words