Task-Aware Representation of Sentences for Generic Text Classification

Proceedings of the 28th International Conference on Computational Linguistics, pages 3202–3213 Barcelona, Spain (Online), December 8-13, 2020 3202 Task-Aware Representation of Sentences for Generic Text Classification Kishaloy Halder† Alan Akbik‡ Josip Krapac† Roland Vollgraf† † Zalando SE ‡ Humboldt-Universit¨at zu Berlin kishaloy.halder@zalando.de alan.akbik@hu-berlin.de josip.krapac@zalando.de roland.vollgraf@zalando.de Abstract State-of-the-art approaches for text classification leverage a transformer architecture with a linear layer on top that outputs a class distribution for a given prediction problem. While effective, this approach suffers from conceptual limitations that affect its utility in few-shot or zero-shot transfer learning scenarios. First, the number of classes to predict needs to be pre-defined. In a transfer learning setting, in which new classes are added to an already trained classifier, all information contained in a linear layer is therefore discarded, and a new layer is trained from scratch. Second, this approach only learns the semantics of classes implicitly from training examples, as opposed to leveraging the explicit semantic information provided by the natural language names of the classes. For instance, a classifier trained to predict the topics of news articles might have classes like “business” or “sports” that themselves carry semantic information. Extending a classifier to predict a new class named “politics” with only a handful of training examples would benefit from both leveraging the semantic information in the name of a new class and using the information contained in the already trained linear layer. This paper presents a novel formulation of text classification that addresses these limitations. It imbues the notion of the task at hand into the transformer model itself by factorizing arbitrary classification problems into a generic binary classification problem. We present experiments in few-shot and zero-shot transfer learning that show that our approach significantly outperforms previous approaches on small training data and can even learn to predict new classes with no training examples at all. The implementation of our model is publicly available at: https://github.com/flairNLP/flair. 1 Introduction Text classification is the task of predicting one or multiple class labels for a given text. It is used in a large number of applications such as spam filtering (Jindal and Liu, 2007), sentiment analysis (Rosenthal et al., 2017), intent detection (Hollerit et al., 2013) or news topic classification (Zhang et al., 2015). The current state-of-the-art approach to text classification leverages a BERT-style transformer architecture (Devlin et al., 2019; Yang et al., 2019; Lan et al., 2020) with a linear classifier layer on top. The transformer is pre-trained on language modelling task, whereas the classifier is randomly initialized. The entire model is then fine-tuned using training examples for all classes, so that the classifier outputs a distribution over all class labels in the prediction problem. This approach is shown to work well, especially if for each class a reasonable amount of training examples is available. Few-shot transfer learning. Real world text classification scenarios are often characterized by a lack of annotated corpora and rapidly changing information needs (Chiticariu et al., 2013), motivating research into methods that allow us to train text classifiers …

Content truncated to 500 words