Recall, Expand, and Multi-Candidate Cross-Encode: Fast and Accurate Ultra-Fine Entity Typing

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 11597–11609 July 9-14, 2023 ©2023 Association for Computational Linguistics Recall, Expand and Multi-Candidate Cross-Encode: Fast and Accurate Ultra-Fine Entity Typing Chengyue Jiang3♠, Wenyang Hui3♠, Yong Jiang♣, Xiaobin Wang♣, Pengjun Xie♣, Kewei Tu♠∗ ♠School of Information Science and Technology, ShanghaiTech University Shanghai Engineering Research Center of Intelligent Vision and Imaging ♣DAMO Academy, Alibaba Group, China {jiangchy,huiwy,tukw}@shanghaitech,edu.cn; {yongjiang.jy,xuanjie.wxb,chengchen.xpj}@alibaba-inc.com Abstract Ultra-fine entity typing (UFET) predicts ex- tremely free-formed types (e.g., president, politician) of a given entity mention (e.g., Joe Biden) in context. State-of-the-art (SOTA) methods use the cross-encoder (CE) based ar- chitecture. CE concatenates a mention (and its context) with each type and feeds the pair into a pretrained language model (PLM) to score their relevance. It brings deeper inter- action between the mention and the type to reach better performance but has to perform N (the type set size) forward passes to infer all the types of a single mention. CE is there- fore very slow in inference when the type set is large (e.g., N = 10k for UFET). To this end, we propose to perform entity typing in a recall-expand-filter manner. The recall and expansion stages prune the large type set and generate K (typically much smaller than N) most relevant type candidates for each mention. At the filter stage, we use a novel model called MCCE to concurrently encode and score all these K candidates in only one forward pass to obtain the final type prediction. We investigate different model options for each stage and con- duct extensive experiments to compare each op- tion, experiments show that our method reaches SOTA performance on UFET and is thousands of times faster than the CE-based architecture. We also found our method is very effective in fine-grained (130 types) and coarse-grained (9 types) entity typing. Our code is avail- able at http://github.com/modelscope/ AdaSeq/tree/master/examples/MCCE. 1 Introduction Ultra-fine entity typing (UFET) (Choi et al., 2018) aims to predict extremely fine-grained types (e.g., president, politician) of a given entity mention within its context. It provides detailed semantic ∗ Kewei Tu is the corresponding author. 3 Equal Contribution. PLM [CLS] Mention & Context [SEP] hcls 0/1 Entailment for PLM [CLS] Mention & Context [SEP] hcls Scores of Type 1 … N Type k (b) Cross-Encoder based methods (a) Multilabel Classification Type k Figure 1: Cross-Encoder and multi-label classification. understandings of entity mentions and is a funda- mental step in fine-grained named entity recogni- tion (Ling and Weld, 2012). It can also be uti- lized to assist various downstream tasks such as relation extraction (Han et al., 2018), keyword ex- traction (Huang et al., 2020) and content recom- mendation (Upadhyay et al., 2021). Most recently, the cross-encoder (CE) based method (Li et al., 2022) achieves the SOTA performance in UFET. Specifically, Li et al. (2022) proposes to treat the mention with its context as a premise, and each ultra-fine-grained type as a hypothesis. They then concatenate them together as input …

Content truncated to 500 words