An Exploration of Three Lightly-supervised Representation Learning Approaches for Named Entity Classification

Proceedings of the 27th International Conference on Computational Linguistics, pages 2312–2324 Santa Fe, New Mexico, USA, August 20-26, 2018. 2312 An Exploration of Three Lightly-supervised Representation Learning Approaches for Named Entity Classification Ajay Nagesh and Mihai Surdeanu University of Arizona, Tucson, AZ, USA {ajaynagesh, msurdeanu}@email.arizona.edu Abstract Several semi-supervised representation learning methods have been proposed recently that mit- igate the drawbacks of traditional bootstrapping: they reduce the amount of semantic drift in- troduced by iterative approaches through one-shot learning; others address the sparsity of data through the learning of custom, dense representation for the information modeled. In this work, we are the first to adapt three of these methods, most of which have been originally proposed for image processing, to an information extraction task, specifically, named entity classifica- tion. Further, we perform a rigorous comparative analysis on two distinct datasets. Our analy- sis yields several important observations. First, all representation learning methods outperform state-of-the-art semi-supervised methods that do not rely on representation learning. To the best of our knowledge, we report the latest state-of-the-art results on the semi-supervised named en- tity classification task. Second, one-shot learning methods clearly outperform iterative repre- sentation learning approaches. Lastly, one of the best performers relies on the mean teacher framework (Tarvainen and Valpola, 2017), a simple teacher/student approach that is independent of the underlying task-specific model. 1 Introduction The paucity of labeled datasets presents tremendous challenges to training machine learning (ML) sys- tems in the real world. Most natural language processing (NLP) tasks are data hungry. The recent success of deep learning for NLP tasks is at least partially due to the fact that such models are exposed to large quantities of labeled data. However, obtaining such rich labeled data is a costly proposition and seldom feasible in many realistic scenarios. Semi-supervised learning (SSL) mitigates the supervision cost by combining a minimal amount of labeled data with a large amount of unlabeled data. In information extraction (IE), SSL often takes the form of bootstrapping, which starts with a few seed examples, e.g., “Barack Obama” as an example of a person’s name, and continues with an iterative approach that alternates between learning extraction patterns such as word n-grams, e.g., the pattern “@ENTITY , former president” could be used to extract person names, and applying these patterns to extract the desired structures (entities, relations, etc.) (Riloff, 1996; Abney, 2007; Carlson et al., 2010b; Gupta and Manning, 2014; Gupta and Manning, 2015, inter alia). There are, however, two drawbacks to this direction. First, as learning in this iterative framework advances, the task often drifts semantically (Komachi et al., 2008) into a related but different space, e.g., from learning women names into learning flower names (McIntosh, 2010; Yangarber, 2003). Second, the statistics used to estimate the quality of the learned patterns are often brittle due to the sparsity of the extraction patterns. Recently, several semi-supervised representation learning methods have been proposed to mitigate these issues. One direction learns custom representations for both patterns and the structures to be extracted …

Content truncated to 500 words