Selective In-Context Data Augmentation for Intent Detection using Pointwise V-Information

Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 1463–1476 May 2-6, 2023 ©2023 Association for Computational Linguistics Selective In-Context Data Augmentation for Intent Detection using Pointwise V-Information Yen-Ting Lin∗ Alexandros Papangelis† Seokhwan Kim† Sungjin Lee† Devamanyu Hazarika† Mahdi Namazifar† Di Jin† Yang Liu† Dilek Hakkani-Tur† ⋆National Taiwan University †Amazon Alexa AI ytl@ieee.org {papangea,seokhwk,sungjinl,dvhaz,mahdinam,djinamzn,yangliud,hakkanit}@amazon.com Abstract This work focuses on in-context data augmenta- tion for intent detection. Having found that aug- mentation via in-context prompting of large pre- trained language models (PLMs) alone does not improve performance, we introduce a novel approach based on PLMs and pointwise V- information (PVI), a metric that can measure the usefulness of a datapoint for training a model. Our method first fine-tunes a PLM on a small seed of training data and then synthesizes new datapoints – utterances that correspond to given intents. It then employs intent-aware fil- tering, based on PVI, to remove datapoints that are not helpful to the downstream intent clas- sifier. Our method is thus able to leverage the expressive power of large language models to produce diverse training data. Empirical results demonstrate that our method can produce syn- thetic training data that achieve state-of-the-art performance on three challenging intent detec- tion datasets under few-shot settings (1.28% absolute improvement in 5-shot and 1.18% ab- solute in 10-shot, on average) and perform on par with the state-of-the-art in full-shot settings (within 0.01% absolute, on average). 1 Introduction Intent detection, defined as the identification of a user’s intent given an utterance, is a fundamental element in task-oriented dialogue systems, usually occurring within the Natural Language Understand- ing (NLU) component. One of the practical chal- lenges of training and deploying NLU modules is data scarcity, due to various reasons, such as under-represented languages, privacy and ethical concerns, or simply the cost of collecting and anno- tating sufficiently large amounts of data for new in- tents. Consequently, accurately identifying intents in limited-resource scenarios has drawn attention from the community (Papangelis et al., 2021; Mehri and Eric, 2021; Zhang et al., 2021b, for example). ∗ Work done during internship at Amazon Alexa AI There are three main families of approaches that address the challenge of limited data for intent de- tection: data augmentation (Peng et al., 2021; Li et al., 2021), focusing on generating high-quality synthetic training and evaluation data; few-shot learning (Zhang et al., 2020, 2021b), focusing on creating learning algorithms that can cope with lim- ited amounts of data; and transfer learning (Namaz- ifar et al., 2021), focusing on learning algorithms that can generalize across domains (therefore not requiring in-domain data). In this work, we fol- low the data augmentation approach, which is a general method that attempts to augment a human- authored dataset with a large set of synthetically- generated instances. Most recent work has sug- gested using Pre-trained Language Models (PLMs) for data augmentations under various setups, e.g., (Peng et al., 2021), showing great improvements in performance. However, simply generating a large number of synthetic …

Content truncated to 500 words