Efficient Named Entity Annotation through Pre-empting

Proceedings of Recent Advances in Natural Language Processing, pages 123–130, Hissar, Bulgaria, Sep 7–9 2015. Efficient Named Entity Annotation through Pre-empting Leon Derczynski University of Sheffield S1 4DP, UK leon@dcs.shef.ac.uk Kalina Bontcheva University of Sheffield S1 4DP, UK kalina@dcs.shef.ac.uk Abstract Linguistic annotation is time-consuming and expensive. One common annotation task is to mark entities – such as names of people, places and organisations – in text. In a document, many segments of text often contain no entities at all. We show that these segments are worth skip- ping, and demonstrate a technique for re- ducing the amount of entity-less text ex- amined by annotators, which we call “pre- empting”. This technique is evaluated in a crowdsourcing scenario, where it pro- vides downstream performance improve- ments for the same size corpus. 1 Introduction Annotating documents is expensive. Given the dominant position of statistical machine learning for many NLP tasks, annotation is unavoidable. It typically requires an expert, but even non-expert annotation work (cf. crowdsourcing) has an as- sociated cost. This makes it important to get the maximum value out of annotation. However, in entity annotation tasks, annotators sometimes are faced with passage of text which bear no entities. These blank examples are espe- cially common outside of the newswire genre, in e.g. social media text (Hu et al., 2013). While finding good examples to annotate next is a prob- lem that has been tackled before, these systems of- ten require a tight feedback loop and great control over which document is presented next. This is not possible in a crowdsourcing scenario, where large volumes of documents need to be presented for annotation simultaneously in order to leverage crowdsourcing’s scalability advantages. The loos- ened feedback loop, and requirement to issue doc- uments in large batches, differentiate the problem scenario from classical active learning. We hypothesise that these blank examples are of limited value as training data for statistical entity annotation systems, and that it is preferable to an- notate texts containing entities over texts without them. This proposition can be evaluated directly, in the context of named entity recognition (NER). If correct, it offers a new pre-annotation task: pre- dicting whether an excerpt of text will contain an entity we are interested in annotating. The goal is to reduce the cost of annotation, or alternatively, to increase the performance of a sys- tem that uses a fixed amount of data. As this pre- annotation task tries to acquire information about entity annotations before they are actually created – specifically, whether or not they exist – we call the task “pre-empting”. Unlike many modern approaches to optimis- ing annotated data, which focus on how to best leverage annotations (perhaps by making infer- ences over those annotations, or by using unla- belled data), we examine the step before this – selecting what to annotate in order to boost later system performance. In this paper, we: • demonstrate that entity-bearing text results in better NER systems; • introduce an entity pre-empting technique; • examine how …

Content truncated to 500 words