Modeling Label Correlations for Ultra-Fine Entity Typing with Neural Pairwise Conditional Random Field

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 6836 - 6847 December 7-11, 2022 ©2022 Association for Computational Linguistics Modeling Label Correlations for Ultra-Fine Entity Typing with Neural Pairwise Conditional Random Field Chengyue Jiang:, Yong Jiang⋆˚, Weiqi Wu, Pengjun Xie⋆, Kewei Tu˚ ⋆DAMO Academy, Alibaba Group, China {jiangchy1997,kewei.tu.nlp}@gmail.com; wuwq1022@foxmail.com; {yongjiang.jy,chengchen.xpj}@alibaba-inc.com Abstract Ultra-fine entity typing (UFET) aims to pre- dict a wide range of type phrases that correctly describe the categories of a given entity men- tion in a sentence. Most recent works infer each entity type independently, ignoring the correlations between types, e.g., when an entity is inferred as a president, it should also be a politician and a leader. To this end, we use an undirected graphical model called pairwise conditional random field (PCRF) to formulate the UFET problem, in which the type variables are not only unarily influenced by the input but also pairwisely relate to all the other type variables. We use various modern backbones for entity typing to compute unary potentials, and derive pairwise potentials from type phrase representations that both capture prior semantic information and facilitate accelerated inference. We use mean-field variational inference for effi- cient type inference on very large type sets and unfold it as a neural network module to enable end-to-end training. Experiments on UFET show that the Neural-PCRF consistently outper- forms its backbones with little cost and results in a competitive performance against cross- encoder based SOTA while being thousands of times faster. We also find Neural-PCRF effec- tive on a widely used fine-grained entity typing dataset with a smaller type set. We pack Neural- PCRF as a network module that can be plugged onto multi-label type classifiers with ease and release it in github.com/modelscope/ adaseq/examples/NPCRF. 1 Introduction Entity typing assigns semantic types to entities mentioned in the text. The extracted type infor- mation has a wide range of applications. It acts as a primitive for information extraction (Yang and Zhou, 2010) and spoken language understanding :This work was done during Chengyue Jiang’s internship at DAMO Academy, Alibaba Group. ˚Yong Jiang and Kewei Tu are corresponding authors. (Coucke et al., 2018), and assists in more com- plicated tasks such as machine reading compre- hension (Joshi et al., 2017) and semantic parsing (Yavuz et al., 2016). During its long history of de- velopment, the granularity of the type set for entity typing and recognition changes from coarse (sized less than 20) (Tjong Kim Sang and De Meulder, 2003; Hovy et al., 2006), to fine (sized around 100) (Weischedel and Brunstein, 2005; Ling and Weld, 2012; Gillick et al., 2014; Ding et al., 2021b), to ultra-fine and free-form (sized 10k) (Choi et al., 2018). The expansion of the type set reveals the diversity of real-world entity categories and the importance of a finer-grained understanding of en- tities in applications (Choi et al., 2018). The increasing number of types results in diffi- culties in predicting correct types, so a better un- derstanding of the entity types and their correla- …

Content truncated to 500 words