Out-of-Domain Detection for Low-Resource Text Classification Tasks

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 3566–3572, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 3566 Out-of-Domain Detection for Low-Resource Text Classification Tasks Ming Tan ∗† Yang Yu ∗† Haoyu Wang ∗† Dakuo Wang ‡ Saloni Potdar † Shiyu Chang ‡ Mo Yu ‡ † IBM Watson ‡ IBM Research Abstract Out-of-domain (OOD) detection for low- resource text classification is a realistic but understudied task. The goal is to detect the OOD cases with limited in-domain (ID) train- ing data, since we observe that training data is often insufficient in machine learning appli- cations. In this work, we propose an OOD- resistant Prototypical Network to tackle this zero-shot OOD detection and few-shot ID classification task. Evaluation on real-world datasets show that the proposed solution out- performs state-of-the-art methods in zero-shot OOD detection task, while maintaining a com- petitive performance on ID classification task. 1 Introduction Text classification tasks in real-world applications often consists of 2 components- In-Doman (ID) classification and Out-of-Domain (OOD) detec- tion components (Liao et al., 2018; Kim and Kim, 2018; Shu et al., 2017; Shamekhi et al., 2018). ID classification refers to classifying a user’s in- put with a label that exists in the training data, and OOD detection refers to designate a special OOD tag to the input when it does not belong to any of the labels in the ID training dataset (Dai et al., 2007). Recent state-of-the-art deep learning (DL) approaches for OOD detection and ID classifica- tion task often require massive amounts of ID or OOD labeled data (Kim and Kim, 2018). In re- ality, many applications have very limited ID la- beled data (i.e., few-shot learning) and no OOD labeled data (i.e., zero-shot learning). Thus, ex- isting methods for OOD detection do not perform well in this setting. One such application is the intent classification for conversational AI services, such as IBM Wat- ∗ Equal contributions from the corresponding authors: {mingtan,yu,wanghaoy}@us.ibm.com. Intent Label Example Help List List what you can help me with. Watson, I need your help Schedule Appointment Can you book a cleaning with my dentist for me? Can you schedule my den- tist’s appointment? End Meeting You can end the meeting now Meeting is over · · · · · · OOD utterances My birthday is coming! blah blah... Table 1: A few-shot ID training set for a conversa- tion service for teleconference management, with OOD testing examples. son Assistant1. For example, Table 1 shows some of the utterances a chat-bot builder provided for training. Each class may only have less than 20 training utterances, due to the high cost of man- ual labelling by domain experts. Meanwhile, the user also expects the service to effectively reject irrelevant queries (as shown at the bottom of Table 1). The challenge of OOD detection is reflected by the undefined in-domain boundary. Although one can provide a certain amount of OOD sam- …

Content truncated to 500 words