Combining Unsupervised Pre-training and Annotator Rationales to Improve Low-shot Text Classification

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 3884–3893, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 3884 Combining Unsupervised Pre-training and Annotator Rationales to Improve Low-shot Text Classification Oren Melamud IBM Research Yorktown Heights, NY, USA oren.melamud@ibm.com Mihaela Bornea IBM Research Yorktown Heights, NY, USA mabornea@us.ibm.com Ken Barker IBM Research Yorktown Heights, NY, USA kjbarker@us.ibm.com Abstract Supervised learning models often perform poorly at low-shot tasks, i.e. tasks for which little labeled data is available for training. One prominent approach for improving low-shot learning is to use unsupervised pre-trained neural models. Another approach is to ob- tain richer supervision by collecting anno- tator rationales (explanations supporting la- bel annotations). In this work, we com- bine these two approaches to improve low- shot text classification with two novel meth- ods: a simple bag-of-words embedding ap- proach; and a more complex context-aware method, based on the BERT model. In ex- periments with two English text classification datasets, we demonstrate substantial perfor- mance gains from combining pre-training with rationales. Furthermore, our investigation of a range of train-set sizes reveals that the sim- ple bag-of-words approach is the clear top performer when there are only a few dozen training instances or less, while more complex models, such as BERT or CNN, require more training data to shine. 1 Introduction When trained on large amounts of labeled data, supervised machine learning models demonstrate impressive performance in various domains rang- ing from text classification to image recognition (Yogatama et al., 2017; Krizhevsky et al., 2012). In many domains, however, labeled data is diffi- cult to obtain. This might be due to annotation cost, or simply because there are no available in- stances to annotate before the system has to make the next classification decision. Imagine, for ex- ample, a personalized email tagger, where new tag types can be created and then used for annotation by the user at any time. When only little training is available, model performance drops dramatically. Various methods have been proposed to opti- mize machine learning models trained on little data with the goal of improving performance for a given train set size or reducing the number of train- ing instances that need to be collected. One of the most prominent for text classification (and other NLP tasks) is transfer learning from unsupervised data, or unsupervised pre-training. This approach has been widely adopted with the introduction of pre-trained word embeddings (Mikolov et al., 2013; Joulin et al., 2017). Word type represen- tations have subsequently been extended with pre- trained, context-dependent representations achiev- ing strong results across a wide range of NLP tasks (Melamud et al., 2016; Peters et al., 2018; Howard and Ruder, 2018). A recent strong performer in this line of research is BERT (Devlin et al., 2018), a multi-layer attention-based neural model that is pre-trained on large amounts of plain text and then fine tuned to specific tasks. Systems using BERT have …

Content truncated to 500 words