Not All Negatives are Equal: Label-Aware Contrastive Loss for Fine-grained Text Classification

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 429–442 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 429 GOLD GOLD GOLD GOLD GOLD: Improving Out-of-Scope Detection in Dialogues using Data Augmentation Derek Chen ASAPP, New York, NY 10007 dchen@asapp.com Zhou Yu Columbia University, NY zy2461@columbia.edu Abstract Practical dialogue systems require robust methods of detecting out-of-scope (OOS) ut- terances to avoid conversational breakdowns and related failure modes. Directly training a model with labeled OOS examples yields rea- sonable performance, but obtaining such data is a resource-intensive process. To tackle this limited-data problem, previous methods focus on better modeling the distribution of in-scope (INS) examples. We introduce GOLD as an orthogonal tech- nique that augments existing data to train better OOS detectors operating in low-data regimes. GOLD generates pseudo-labeled candidates using samples from an auxiliary dataset and keeps only the most beneficial can- didates for training through a novel filtering mechanism. In experiments across three tar- get benchmarks, the top GOLD model outper- forms all existing methods on all key metrics, achieving relative gains of 52.4%, 48.9% and 50.3% against median baseline performance. We also analyze the unique properties of OOS data to identify key factors for optimally apply- ing our proposed method.1 1 Introduction Detecting out-of-scope scenarios is an essential skill of dialogue systems deployed into the real world. While an ideal system would behave ap- propriately in all conversational settings, such per- fection is not possible given that training data is finite, while user inputs are not (Geiger et al., 2019). Out-of-distribution issues occur when the model encounters situations not covered during training, including novel user intents, domain shifts or custom entities (Kamath et al., 2020; Cavalin et al., 2020). Unique to conversations, dialogue breakdowns represent cases where the user cannot continue the interaction with the system, perhaps 1All code and data for major experiments are available at https://github.com/asappresearch/gold Generated OOS Labels from Data Augmentation Small seed set composed of OOS examples Pool of source utterances Figure 1: GOLD performs data augmentation by ex- tracting utterances from a source dataset and merging those sentences with known OOS samples from the tar- get dataset to generate pseudo-labeled OOS examples. due to ambiguous requests or prior misunderstand- ings (Martinovsky and Traum, 2003; Higashinaka et al., 2016). Such breakdowns might fall within the distribution of plausible utterances, yet still fail to make sense due to the given context. OOS detection aims to recognize both out-of-distribution problems and dialogue breakdowns. Prior methods tackling OOS detection in text have shown great promise, but typically assume access to a sufficient amount of labeled OOS data during training (Larson et al., 2019), which is unre- alistic in open-world settings (Fei and Liu, 2016). Alternative methods have also been explored which train a supporting model using in-scope data rather than directly training a core model to detect OOS instances (Gangal et al., 2020). As a result, they 430 suffer from a mismatch where the objective during training does not line up with the …

Content truncated to 500 words