The Benefits of Word Embeddings Features for Active Learning in Clinical Information Extraction

The Benefits of Word Embeddings Features for Active Learning in Clinical Information Extraction Mahnoosh Kholghi, Lance De Vine, Laurianne Sitbon, Guido Zuccon Queensland University of Technology {m1.kholghi, l.devine, laurianne.sitbon, g.zuccon}@qut.edu.au Anthony Nguyen The Australian e-Health Research Centre, CSIRO anthony.nguyen@csiro.au Abstract This study investigates the use of unsu- pervised word embeddings and sequence features for sample representation in an active learning framework built to extract clinical concepts from clinical free text. The objective is to further reduce the manual annotation effort while achieving higher effectiveness compared to a set of baseline features. Unsupervised features are derived from skip-gram word embed- dings and a sequence representation ap- proach. The comparative performance of unsupervised features and baseline hand- crafted features in an active learning framework are investigated using a wide range of selection criteria including least confidence, information diversity, infor- mation density and diversity, and domain knowledge informativeness. Two clinical datasets are used for evaluation: the i2b2/VA 2010 NLP challenge and the ShARe/CLEF 2013 eHealth Evaluation Lab. Our results demonstrate significant improvements in terms of effectiveness as well as annotation effort savings across both datasets. Using unsupervised features along with baseline features for sample representation lead to further sav- ings of up to 9% and 10% of the token and concept annotation rates, respective- ly. 1 Introduction Active learning (AL) has recently received con- siderable attention in clinical information extrac- tion, as it promises to automatically annotate clinical free text with less manual annotation ef- fort than supervised learning approaches, while achieving the same effectiveness (Boström & Dalianis, 2012; Chen et al., 2015; Chen et al., 2012; Figueroa et al., 2012; Kholghi et al., 2015, 2016; Ohno-Machado et al., 2013). Active learn- ing is particularly important in the clinical do- main because of the costs incurred in preparing high quality annotated data as required by super- vised machine learning approaches for a wide range of data analysis applications such as re- trieving, reasoning, and reporting. Active learn- ing is a human-in-the-loop process in which at each iteration, a set of informative instances is automatically selected by a query strategy (Settles, 2012) and annotated in order to re-train or update the supervised model (see Figure 1). The query strategy, as a key component of the AL process, plays an important role in the per- formance of AL approaches. Figure 1. Active learning process. The learning models at each iteration are typi- cally built using supervised learning algorithms. The associated output of the learning model (i.e. the posterior probability) is usually leveraged in identifying and selecting the next set of informa- tive instances. Hence, it is important to build ac- Mahnoosh Kholghi, Lance De Vine, Laurianne Sitbon, Guido Zuccon and Anthony Nguyen. 2016. The Benefits of Word Embeddings Features for Active Learning in Clinical Information Extraction. In Proceedings of Australasian Language Technology Association Workshop, pages 25−34. curate statistical models early in the process, and at each iteration. Previous studies have high- lighted that the feature set, which is used to rep- resent data instances, is …

Content truncated to 500 words