Modeling Fine-Grained Entity Types with Box Embeddings

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 2051–2064 August 1–6, 2021. ©2021 Association for Computational Linguistics 2051 Modeling Fine-Grained Entity Types with Box Embeddings Yasumasa Onoe♠, Michael Boratko♢, Andrew McCallum♢♣, Greg Durrett♠ ♠The University of Texas at Austin ♢University of Massachusetts Amherst ♣Google Research {yasumasa,gdurrett}@cs.utexas.edu {mboratko,mccallum}@cs.umass.edu mccallum@google.com Abstract Neural entity typing models typically repre- sent fine-grained entity types as vectors in a high-dimensional space, but such spaces are not well-suited to modeling these types’ com- plex interdependencies. We study the ability of box embeddings, which embed concepts as d-dimensional hyperrectangles, to capture hi- erarchies of types even when these relation- ships are not defined explicitly in the ontol- ogy. Our model represents both types and en- tity mentions as boxes. Each mention and its context are fed into a BERT-based model to embed that mention in our box space; essen- tially, this model leverages typological clues present in the surface text to hypothesize a type representation for the mention. Box con- tainment can then be used to derive both the posterior probability of a mention exhibiting a given type and the conditional probability rela- tions between types themselves. We compare our approach with a vector-based typing model and observe state-of-the-art performance on several entity typing benchmarks. In addi- tion to competitive typing performance, our box-based model shows better performance in prediction consistency (predicting a supertype and a subtype together) and confidence (i.e., calibration), demonstrating that the box-based model captures the latent type hierarchies bet- ter than the vector-based model does.1 1 Introduction The development of named entity recognition and entity typing has been characterized by a growth in the size and complexity of type sets: from 4 (Tjong Kim Sang and De Meulder, 2003) to 17 (Hovy et al., 2006) to hundreds (Weischedel and Brun- stein, 2005; Ling and Weld, 2012) or thousands (Choi et al., 2018). These types follow some kind 1The code is available at https://github.com/ yasumasaonoe/Box4Types. of hierarchical structure (Weischedel and Brunstein, 2005; Ling and Weld, 2012; Gillick et al., 2014; Murty et al., 2018), so effective models for these tasks frequently engage with this hierarchy explic- itly. Prior systems incorporate this structure via hierarchical losses (Murty et al., 2018; Xu and Barbosa, 2018; Chen et al., 2020) or by embed- ding types into a high-dimensional Euclidean or hyperbolic space (Yogatama et al., 2015; L´opez and Strube, 2020). However, the former approach re- quires prior knowledge of the type hierarchy, which is unsuitable for a recent class of large type sets where the hierarchy is not explicit (Choi et al., 2018; Onoe and Durrett, 2020a). The latter ap- proaches, while leveraging the inductive bias of hyperbolic space to represent trees, lack a proba- bilistic interpretation of the embedding and do not naturally capture all of the complex type relation- ships beyond strict containment. In this paper, we describe an approach that rep- resents entity types with box embeddings in a high- …

Content truncated to 500 words