Analysing recall loss in named entity slot filling

Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 820–830, October 25-29, 2014, Doha, Qatar. c⃝2014 Association for Computational Linguistics Analysing recall loss in named entity slot filling Glen Pink Joel Nothman James R. Curran e -lab, School of Information Technologies University of Sydney NSW 2006, Australia {glen.pink,joel.nothman,james.r.curran}@sydney.edu.au Abstract State-of-the-art fact extraction is heavily constrained by recall, as demonstrated by recent performance in TAC Slot Filling. We isolate this recall loss for NE slots by systematically analysing each stage of the slot filling pipeline as a filter over correct answers. Recall is critical as candidates never generated can never be recovered, whereas precision can always be increased in downstream processing. We provide precise, empirical confirma- tion of previously hypothesised sources of recall loss in slot filling. While NE type constraints substantially reduce the search space with only a minor recall penalty, we find that 10% to 39% of slot fills will be entirely ignored by most systems. One in six correct answers are lost if coreference is not used, but this can be mostly retained by simple name matching rules. 1 Introduction The TAC Knowledge Base Population (KBP) Slot Filling (SF) consists of extracting named attributes from text. Given a query, e.g. John Kerry, a system searches a corpus for documents which contain the entity. It then fills a list of slots, named attributes such as (per:spouse, Teresa Heinz). The top TAC SF 2013 (TAC13) system scored 37.3% F-score (Roth et al., 2013), and the median F-score was 16.9% (Surdeanu, 2013). Recall for SF systems is especially low, with many systems using precise extractors with low recall. Precision ranges from 9% to 40% greater than recall for the top 5 systems in TAC13, and unsurprisingly, Roth et al. (2013) has the highest recall at 33%. Closing the recall gap without substantially increasing the search space is critical to improving SF results. Ji and Grishman (2011) and Min and Grishman (2012) identify many of the challenges of SF, and suggest that inference, coreference and named en- tity recognition (NER) are key sources of error. Min and Grishman categorise the slot fills found by human annotators but not found in the aggre- gated output of all systems. However, this ap- proach only allows them to hypothesise the likely source of recall loss. For instance, it is impossible to distinguish candidate generation errors from an- swer merging errors. Roth et al. (2014) categorise these errors at a high level, without specific anal- ysis of candidate generation pipeline components such as coreference. In this paper, we take this analysis further by performing a systematic recall analysis that al- lows us to pinpoint the cause of every recall er- ror (candidates lost that can never be recovered) and estimate upper bounds on recall in existing ap- proaches. We implement a collection of na¨ıve SF systems utilizing a set of increasingly restrictive filters over documents and named entities (NEs). TAC has three slot types: NE, string and value slots. We consider …

Content truncated to 500 words