Systematic Study of Long Tail Phenomena in Entity Linking

Proceedings of the 27th International Conference on Computational Linguistics, pages 664–674 Santa Fe, New Mexico, USA, August 20-26, 2018. 664 Systematic Study of Long Tail Phenomena in Entity Linking Filip Ilievski, Piek Vossen, Stefan Schlobach Vrije Universiteit Amsterdam De Boelelaan 1105, 1081 HV Amsterdam The Netherlands {f.ilievski, piek.vossen, k.s.schlobach}@vu.nl Abstract State-of-the-art entity linkers achieve high accuracy scores with probabilistic methods. However, these scores should be considered in relation to the properties of the datasets they are evaluated on. Until now, there has not been a systematic investigation of the properties of entity linking datasets and their impact on system performance. In this paper we report on a series of hypotheses regarding the long tail phenomena in entity linking datasets, their interaction, and their impact on system performance. Our systematic study of these hypotheses shows that evaluation datasets mainly capture head entities and only incidentally cover data from the tail, thus encouraging systems to overfit to popular/frequent and non-ambiguous cases. We find the most difficult cases of entity linking among the infrequent candidates of ambiguous forms. With our findings, we hope to inspire future designs of both entity linking systems and evaluation datasets. To support this goal, we provide a list of recommended actions for better inclusion of tail cases. 1 Introduction The task of Entity Linking (EL) anchors recognized entity mentions in text to their semantic representa- tion, thus establishing identity and facilitating the exploitation of background knowledge, easy integra- tion, and comparison and reuse of systems. The past years featured a plethora of EL systems: DBpedia Spotlight (Daiber et al., 2013), WAT (Piccinno and Ferragina, 2014), AGDISTIS MAG (Moussallem et al., 2017), to name a few. These systems propose various probabilistic algorithms for graph optimization or machine learning, in order to perform disambiguation, i.e., to pick the correct entity candidate for a surface form in a given context. The reported accuracy scores are fairly high, which gives an impression that the task of EL is both well-understood and fairly solved by existing systems. At the same time, several papers (Ilievski et al., 2016; Van Erp et al., 2016; Esquivel et al., 2017; Ilievski et al., 2017) have argued that state-of-the-art EL systems base their performance on frequent ‘head’ cases, while performance drops significantly when moving towards the rare ‘long tail’ entities. This statement seems intuitively obvious, but no previous work has quantified what the ‘head’ and ‘tail’ of EL entails. In fact, the lack of definition of head and tail in this task prevents the (in)validation of the hypothesis that interpreting some (classes of) cases is more challenging for systems than others. This, in turn, means that we are currently unable to identify the difficult cases of EL for which current systems need to be adapted, or new approaches need to be developed. Previous linguistic studies which analyze words distributions can not be applied for this purpose, because they do not study reference, nor the relation of the head-tail distribution to system performance. Understanding the tail cases better and …

Content truncated to 500 words