ConVEx: Data-Efficient and Few-Shot Slot Labeling

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3375–3389 June 6–11, 2021. ©2021 Association for Computational Linguistics 3375 ConVEx: Data-Efficient and Few-Shot Slot Labeling Matthew Henderson1 and Ivan Vuli´c1,2 1 PolyAI Ltd, London, UK 2 Language Technology Lab, University of Cambridge, UK ivan@polyai.com Abstract We propose ConVEx (Conversational Value Extractor), an efficient pretraining and fine- tuning neural approach for slot-labeling dia- log tasks. Instead of relying on more general pretraining objectives from prior work (e.g., language modeling, response selection), Con- VEx’s pretraining objective, a novel pairwise cloze task using Reddit data, is well aligned with its intended usage on sequence labeling tasks. This enables learning domain-specific slot labelers by simply fine-tuning decoding layers of the pretrained general-purpose se- quence labeling model, while the majority of the pretrained model’s parameters are kept frozen. We report state-of-the-art performance of ConVEx across a range of diverse domains and data sets for dialog slot-labeling, with the largest gains in the most challenging, few-shot setups. We believe that ConVEx’s reduced pretraining times (i.e., only 18 hours on 12 GPUs) and cost, along with its efficient fine- tuning and strong performance, promise wider portability and scalability for data-efficient sequence-labeling tasks in general. 1 Introduction Slot labeling or slot filling is a critical natural language understanding (NLU) component of any task-oriented dialog system (Young, 2002, 2010; Tür and De Mori, 2011, inter alia). Its goal is to fill the correct values associated with predefined slots: e.g., a dialog system for restaurant bookings is expected to fill slots such as date, time, and the number of guests with the values extracted from a user utterance (e.g., next Thursday, 7pm, 4 people). Setting up task-oriented dialog systems, as well as slot labeling methods in particular, to sup- port new tasks and domains is highly challeng- ing due to inherent scarcity of expensive expert- annotated data for a plethora of intended use sce- narios (Williams, 2014; Henderson et al., 2014; Budzianowski et al., 2018; Zhao et al., 2019). One plausible and promising solution is the creation of data-efficient models that learn from only a hand- ful annotated examples in few-shot scenarios. This approach has been shown promising for learning in- tent detectors (Casanueva et al., 2020; Krone et al., 2020; Bunk et al., 2020) as well as for slot-filling methods (Hou et al., 2020; Coope et al., 2020). The dominant paradigm followed by the existing models of few-shot slot labeling is transfer learn- ing (Ruder et al., 2019): 1) they rely on representa- tions from models pretrained on large data collec- tions in a self-supervised manner on some general NLP tasks such as (masked) language modeling (Devlin et al., 2019; Conneau et al., 2020; Brown et al., 2020) or response selection (Henderson et al., 2019b, 2020; Cer et al., 2018); and then 2) add ad- ditional task-specific layers for modeling the input sequences. However, we detect several gaps with the existing setup, and set to address them in …

Content truncated to 500 words