Solving the AL Chicken-and-Egg Corpus and Model Problem: Model-free Active Learning for Phenomena-driven Corpus Construction

Solving the AL Chicken-and-Egg Corpus and Model Problem: Model-free Active Learning for Phenomena-driven Corpus Construction Dain Kaplan, Neil Rubens, Simone Teufel, Takenobu Tokunaga Tokyo Institute of Technology, Stanford University, University of Cambridge, Tokyo Institute of Technology Dept. of Computer Science, mediaX / H-STAR, Computer Laboratory, Dept. of Computer Science JAPAN, USA, UK, JAPAN dain@cl.cs.titech.ac.jp, rubens@activeintel.org, simone.teufel@cl.cam.ac.uk, take@cl.cs.titech.ac.jp Abstract Active learning (AL) is often used in corpus construction (CC) for selecting “informative” documents for annotation. This is ideal for focusing annotation efforts when all documents cannot be annotated, but has the limitation that it is carried out in a closed-loop, selecting points that will improve an existing model. For phenomena-driven and exploratory CC, the lack of existing-models and specific task(s) for using it make traditional AL inapplicable. In this paper we propose a novel method for model-free AL utilising characteristics of phenomena for applying AL to select documents for annotation. The method can also supplement traditional closed-loop AL-based CC to broaden the utility of the corpus created beyond a single task. We introduce our tool, MOVE, and show its potential with a real world case-study. Keywords: corpus construction, active learning, tools 1. Introduction In recent years, we have seen an explosion of supervised machine learning (ML) techniques in the NLP commu- nity. The accuracy of these supervised ML methods largely depends on three factors: (1) the quality of the prediction algorithm, (2) the quality of training data, and (3) the quan- tity of training data. While research often concentrates on improving prediction algorithms (i.e. creating/improving models), even the best algorithms will fail if they are fed poor quality data during training, i.e. “garbage in, garbage out”. On the other end, corpus construction (CC) typi- cally faces the challenge of maximising the usefulness of annotation while keeping the costs within the allotted bud- get. Corpus annotation is usually undergone in one of three ways: (1) purely automatic, (2) automated annotation fol- lowed by manual correction, and (3) purely manual anno- tation (McEnery et al., 1995). As active learning (AL) for CC identifies the most “infor- mative” data for annotation (Olsson, 2009; Tomanek and Olsson, 2009; Settles, 2009; Song and Yao, 2010), at first glance it seems like a perfect fit when by necessity you cannot annotate all documents. For example, consider Fig- ure 1, which shows random input vs. input selected using AL to consider the network structure. However, this brings us to what we define as the chicken- and-egg corpus and model conundrum, which refers to how AL often happens in a closed-loop process, the underlying model or models directly influencing which data is selected for annotation, which improves the model’s accuracy, and so on. For exploratory CC in which creators may be investigating hitherto under/unexplored phenomena, no existing corpora and no existing models preclude the use of AL entirely – there is no ‘loop’ yet to close. Further, even for closed-loop AL-based CC, the resultant corpora may have very limited or specific use. In this paper we propose an active …

Content truncated to 500 words