Robust Text Classification for Sparsely Labelled Data Using Multi-level Embeddings

Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 2333–2343, Osaka, Japan, December 11-17 2016. Robust Text Classification for Sparsely Labelled Data Using Multi-level Embeddings Simon Baker 1,2 Douwe Kiela 1 Anna Korhonen 2 1Computer Laboratory, 15 JJ Thomson Avenue 2 Language Technology Lab, DTAL University of Cambridge, UK {sb895|dk427|alk23}@cam.ac.uk Abstract The conventional solution for handling sparsely labelled data is extensive feature engineering. This is time consuming and task and domain specific. We present a novel approach for learning embedded features that aims to alleviate this problem. Our approach jointly learns embeddings at different levels of granularity (word, sentence and document) along with the class labels. The intuition is that topic semantics represented by embeddings at multiple levels results in better classification. We evaluate this approach in unsupervised and semi-supervised settings on two sparsely labelled classification tasks, outperforming the handcrafted models and several embed- ding baselines. 1 Introduction The objective of text classification is to label a scope of text according to predefined labels. While general domains tend to have sufficient amounts of labelled data, in specialised domains (e.g., scientific literature) such data are often scarce and labelled instances number in the hundreds, or low thousands at most. Such domains may also require highly specialised annotators, making labelled data expensive and difficult to obtain (Simpson and Demner-Fushman, 2012). In order to mitigate the data sparsity problem, a lot of handcrafting is needed to engineer features specific to the task and domain. Typically this process involves a long NLP pipeline, e.g., POS-tagging, parsing, named entity recognition, semantic role labelling, feature selection, etc. Consequently, ap- proaches based on handcrafting can be prohibitively time consuming, and since the resultant features are domain dependent, these systems are difficult to port to other domains (Sebastiani, 2002; Dai et al., 2007). While unsupervised and lightly-supervised methods can bypass the need for labelled data, they in turn tend to suffer from lower performance (Zhang and Elhadad, 2013; Quan et al., 2014; Aggarwal and Zhai, 2012). In this paper, we present a novel approach to text classification that is especially beneficial in situations were labelled datasets are small. Our approach builds on the Distributed Memory (DM) model by Le and Mikolov (2014). The fast and simple unsupervised DM model acquires paragraph level embeddings. We improve on the model so that we jointly learn multi-level embeddings that encode class-label topical information in addition to text. We jointly learn a model that captures embedding representation for the target class labels, as well as word-, sentence- and document-level representations in the same space. From these multi-level em- beddings we derive a set of features. Our approach requires no manual feature engineering, can cope with small amounts of labelled data and produces features that are more robust to domain variation and portable across domains. At the document-level, the overall “topic” is a mixture of the sub-topics of paragraphs in that doc- ument. The topics of the paragraphs are in turn mixtures of the sentence topics, all the …

Content truncated to 500 words