Benchmarking Zero-shot Text Classification: Datasets, Evaluation and Entailment Approach

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 3914–3923, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 3914 Benchmarking Zero-shot Text Classification: Datasets, Evaluation and Entailment Approach Wenpeng Yin, Jamaal Hay, Dan Roth Cognitive Computation Group Department of Computer and Information Science, University of Pennsylvania {wenpeng,jamaalh,danroth}@seas.upenn.edu Abstract Zero-shot text classification (0SHOT-TC) is a challenging NLU problem to which little at- tention has been paid by the research com- munity. 0SHOT-TC aims to associate an ap- propriate label with a piece of text, irrespec- tive of the text domain and the aspect (e.g., topic, emotion, event, etc.) described by the label. And there are only a few articles study- ing 0SHOT-TC, all focusing only on topical categorization which, we argue, is just the tip of the iceberg in 0SHOT-TC. In addition, the chaotic experiments in literature make no uni- form comparison, which blurs the progress. This work benchmarks the 0SHOT-TC problem by providing unified datasets, standardized evaluations, and state-of-the-art baselines. Our contributions include: i) The datasets we pro- vide facilitate studying 0SHOT-TC relative to conceptually different and diverse aspects: the “topic” aspect includes “sports” and “politics” as labels; the “emotion” aspect includes “joy” and “anger”; the “situation” aspect includes “medical assistance” and “water shortage”. ii) We extend the existing evaluation setup (label- partially-unseen) – given a dataset, train on some labels, test on all labels – to include a more challenging yet realistic evaluation label-fully-unseen 0SHOT-TC (Chang et al., 2008), aiming at classifying text snippets with- out seeing task specific training data at all. iii) We unify the 0SHOT-TC of diverse aspects within a textual entailment formulation and study it this way. 1 1 Introduction Supervised text classification has achieved great success in the past decades due to the availability of rich training data and deep learning techniques. However, zero-shot text classification (0SHOT-TC) 1 https://cogcomp.seas.upenn.edu/page/ publication_view/883 The plague in Mongolia, occurring last week, has caused more than a thousand isolation health, finance, politics, sports, etc. anger, joy, sadness, fear etc. shelter, water, medical assistance, etc. news, serious etc. more possible labels "topic" aspect "emotion" aspect "situation" aspect Figure 1: A piece of text can be assigned labels which describe the different aspects of the text. Positive labels are in blue. has attracted little attention despite its great po- tential in real world applications, e.g., the intent recognition of bank consumers. 0SHOT-TC is chal- lenging because we often have to deal with classes that are compound, ultra-fine-grained, changing over time, and from different aspects such as topic, emotion, etc. Existing 0SHOT-TC studies have mainly the fol- lowing three problems. First problem. The 0SHOT-TC problem was modeled in a too restrictive vision. Firstly, most work only explored a single task, which was mainly topic categorization, e.g., (Pushp and Sri- vastava, 2017; Yogatama et al., 2017; Zhang et al., 2019). We argue that this is only the tiny tip of the iceberg for 0SHOT-TC. Secondly, there …

Content truncated to 500 words