EntEval: A Holistic Evaluation Benchmark for Entity Representations

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 421–433, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 421 EntEval: A Holistic Evaluation Benchmark for Entity Representations Mingda Chen3∗ Zewei Chu1∗ Yang Chen2 Karl Stratos4† Kevin Gimpel3 1University of Chicago, IL, USA 2Ohio State University, OH, USA 3Toyota Technological Institute at Chicago, IL, USA 4Rutgers University, NJ, USA {mchen,kgimpel}@ttic.edu,stratos@cs.rutgers.edu,zeweichu@uchicago.edu,chen.9279@osu.edu Abstract Rich entity representations are useful for a wide class of problems involving entities. De- spite their importance, there is no standard- ized benchmark that evaluates the overall qual- ity of entity representations. In this work, we propose EntEval: a test suite of diverse tasks that require nontrivial understanding of enti- ties including entity typing, entity similarity, entity relation prediction, and entity disam- biguation. In addition, we develop training techniques for learning better entity represen- tations by using natural hyperlink annotations in Wikipedia. We identify effective objectives for incorporating the contextual information in hyperlinks into state-of-the-art pretrained lan- guage models (Peters et al., 2018a) and show that they improve strong baselines on multiple EntEval tasks.1 1 Introduction Entity representations play a key role in numer- ous important problems including language mod- eling (Ji et al., 2017), dialogue generation (He et al., 2017), entity linking (Gupta et al., 2017), and story generation (Clark et al., 2018). One suc- cessful line of work on learning entity representa- tions has been learning static embeddings: that is, assign a unique vector to each entity in the train- ing data (Gupta et al., 2017; Yamada et al., 2016, 2017). While these embeddings are useful in many applications, they have the obvious drawback of not accommodating unknown entities. Another limiting factor is the lack of an evaluation bench- mark: it is often difficult to know which entity rep- resentations are better for which tasks. ∗Equal contribution. Listed in alphabetical order. †Work done while the author was at Toyota Technologi- cal Institute at Chicago. 1Data processing and evaluation scripts are available at https://github.com/ZeweiChu/EntEval We introduce EntEval: a carefully designed benchmark for holistically evaluating entity repre- sentations. It is a test suite of diverse tasks that re- quire nontrivial understanding of entities, includ- ing entity typing, entity similarity, entity relation prediction, and entity disambiguation. Motivated by the recent success of contextualized word rep- resentations (henceforth: CWRs) from pretrained models (McCann et al., 2017; Peters et al., 2018a; Devlin et al., 2018; Yang et al., 2019; Liu et al., 2019b), we propose to encode the mention context or the description to dynamically represent an en- tity. In addition, we perform an in-depth compar- ison of ELMo and BERT-based embeddings and find that they show different characteristics on dif- ferent tasks. We analyze each layer of the CWRs and make the following observations: • The dynamically encoded entity representations show a strong improvement on the entity dis- ambiguation task compared to prior work using static entity embeddings. • BERT-based entity representations require fur- ther …

Content truncated to 500 words