A Practical Incremental Learning Framework For Sparse Entity Extraction

Proceedings of the 27th International Conference on Computational Linguistics, pages 700–710 Santa Fe, New Mexico, USA, August 20-26, 2018. 700 A Practical Incremental Learning Framework For Sparse Entity Extraction Hussein S. Al-Olimat1∗, Steven Gustafson2, Jason Mackay3 Krishnaprasad Thirunarayan1, and Amit Sheth1 1 Kno.e.sis Center, Wright State University, Dayton, OH {hussein;tkprasad;amit}@knoesis.org 2 Maana Inc, Bellevue, WA steven.gustafson@maana.io 3 GoDaddy Inc, Kirkland, WA jmackay@godaddy.com Abstract This work addresses challenges arising from extracting entities from textual data, including the high cost of data annotation, model accuracy, selecting appropriate evaluation criteria, and the overall quality of annotation. We present a framework that integrates Entity Set Expansion (ESE) and Active Learning (AL) to reduce the annotation cost of sparse data and provide an online evaluation method as feedback. This incremental and interactive learning framework allows for rapid annotation and subsequent extraction of sparse data while maintaining high accuracy. We evaluate our framework on three publicly available datasets and show that it drastically re- duces the cost of sparse entity annotation by an average of 85% and 45% to reach 0.9 and 1.0 F-Scores respectively. Moreover, the method exhibited robust performance across all datasets. 1 Introduction Entity extraction methods delimit all mentions of an entity class (e.g., Location Names, Proteins, or Auto Parts IDs) in corpora of unstructured text. Supervised machine learning approaches have been effective in extracting such entity mentions, such as (Finkel et al., 2005). However, the prohibitive cost of obtaining a large volume of labeled data for training makes them unattractive and hard to use in realistic settings where resources may be scarce or costly to create (e.g., aviation engineer needing to annotate engine maintenance data). In this paper, we address the problem by developing a practical solution exploiting the strengths of the supervised sequence labeling techniques (Ye et al., 2009) while reducing the high cost of annotation. Training data and the annotation approach are the two focal points of any supervised method. A model can be built from pre-annotated data or using de novo annotations. There are many pre-annotated and publicly available corpora such as CoNLL-2003 (Tjong Kim Sang and De Meulder, 2003), GENIA (Ohta et al., 2001), and BioCreative II (et. al., 2008). However, having such data leaves us with two chal- lenges: (i) no training dataset can be found for many of the domain-specific entity classes in enterprise corpora due to data privacy concerns, IP restrictions, and problem specific use cases (needing de novo annotations), and (ii) data sparsity, which hinders the performance of supervised models on unseen data (needing incrementally augmented annotations to the annotated sentence pool). We propose a practical solution to model building through (i) rapid auto-annotation, to create models with reduced cost, and (ii) flexible stopping criteria using an online evaluation method, which calculates the confidence of a model on unseen data without a need for a gold standard. Having such an online feedback method supports incremental learning which allows us to partially overcome the data sparsity problem (see Section 2). Mainstream entity annotation approaches typically …

Content truncated to 500 words