ManyEnt: A Dataset for Few-shot Entity Typing

Proceedings of the 28th International Conference on Computational Linguistics, pages 5553–5557 Barcelona, Spain (Online), December 8-13, 2020 5553 ManyEnt: A Dataset for Few-shot Entity Typing Markus Eberts Kevin Pech Adrian Ulges RheinMain University of Applied Sciences, Germany {markus.eberts, adrian.ulges}@hs-rm.de kevin.pech@student.hs-rm.de Abstract We introduce ManyEnt, a benchmark for entity typing models in few-shot scenarios. ManyEnt offers a rich typeset, with a fine-grain variant featuring 256 entity types and a coarse-grain one with 53 entity types. Both versions have been derived from the Wikidata knowledge graph in a semi-automatic fashion. We also report results for two baselines using BERT, reaching up to 70.68% accuracy (10-way 1-shot). 1 Introduction Information extraction is targeted at inferring knowledge from unstructured documents, most commonly in the form of entities and relations between them. Like many other NLP tasks, the field has recently experienced a boost from deep neural networks (Wang et al., 2019; Wadden et al., 2019; Li et al., 2019). While research datasets are usually derived from Wikipedia, for industrial applications it is the adaptation to new domains which matters: Here, different relations may be of interest than on the source domain, and existing relations may express themselves differently (think of the “part of” relation, which differs between the medical domain and mechanical engineering). As data is often scarce on the target domain, recent work has addressed few-shot scenarios: A model is pre-trained on a training set of relations representing the source domain and is then adapted to new relations representing the target domain, usually using only few (i.e., 1−5) samples. One prominent example is the FewRel benchmark (Han et al., 2018) for relation classification: Given a sentence in which two entity mentions are marked, the tasks is to assign the entity pair to a relation. Models are first trained on a dataset of 80 relation types and are then applied to held-out test relation types. While FewRel addresses relations, we argue that – since information extraction models targeted at new domains will have to adapt both to new entities and relations – the few-shot scenario is also interesting for entity typing. This way, models pretrained on some entity types (such as computer) can be adapted to other types (such as smartphone). Therefore, we suggest a novel benchmark “ManyEnt”1 for few- shot entity typing. The benchmark is based on FewRel and features a similar setting: Given a sentence with a highlighted entity, the entity is to be assigned to an entity type. Here, a rich number of types is beneficial: Only if a breadth of types is known in training transfer can be expected to work well by adapting to “similar” types as the ones from the source domain. While FewRel offers a much richer typeset of relations (80 types) than other datasets, e.g. (Hendrickx et al., 2010), it offers no entity types. Therefore, our first contribution is a semi-automatic annotation of entities on FewRel at two granularities (53 types and 256 types) using the Wikidata knowledge graph. Second, we also report the results of …

Content truncated to 500 words