Does Your Model Classify Entities Reasonably? Diagnosing and Mitigating Spurious Correlations in Entity Typing

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 8642 - 8658 December 7-11, 2022 ©2022 Association for Computational Linguistics Does Your Model Classify Entities Reasonably? Diagnosing and Mitigating Spurious Correlations in Entity Typing Nan Xu, Fei Wang, Bangzheng Li, Mingtao Dong, Muhao Chen Department of Computer Science & Information Sciences Institute University of Southern California {nanx,fwang598,bangzhen,mingtaod,muhaoche}@usc.edu Abstract Entity typing aims at predicting one or more words that describe the type(s) of a specific mention in a sentence. Due to shortcuts from surface patterns to annotated entity labels and biased training, existing entity typing models are subject to the problem of spurious correla- tions. To comprehensively investigate the faith- fulness and reliability of entity typing methods, we first systematically define distinct kinds of model biases that are reflected mainly from spurious correlations. Particularly, we identify six types of existing model biases, including mention-context bias, lexical overlapping bias, named entity bias, pronoun bias, dependency bias, and overgeneralization bias. To mitigate model biases, we then introduce a counterfac- tual data augmentation method. By augment- ing the original training set with their debiased counterparts, models are forced to fully compre- hend sentences and discover the fundamental cues for entity typing, rather than relying on spurious correlations for shortcuts. Experimen- tal results on the UFET dataset show our coun- terfactual data augmentation approach helps im- prove generalization of different entity typing models with consistently better performance on both the original and debiased test sets1. 1 Introduction Given a sentence with an entity mention, the entity typing task aims at predicting one or more words or phrases that describe the type(s) of that spe- cific mention (Ling and Weld, 2012; Gillick et al., 2014; Choi et al., 2018). This task essentially sup- ports the structural perception of unstructured text (Distiawan et al., 2019), being an important step for natural language understanding (NLU). More specifically, entity typing has a broad impact on var- ious NLP tasks that depend on type understanding, including coreference resolution (Onoe and Dur- rett, 2020), entity linking (Hou et al., 2020; Tianran 1Code and resources are available at https:// github.com/luka-group/DiagnoseET. Input: Last week I stayed in Treasure Island for two nights when visiting Las Vegas. Gold labels: hotel, resort, location, place Pred labels: island, land, location, place Input: Next day (-> Next twenty-four hour period), after the Slovaks captured Rymanow Zdroj and the the Germans seized Krosno, the Brigade was ordered to withdraw to Sanok and leave Dukla. Gold labels: day, time, event, date, year Pred labels: day, time, date -> time, hour period Input: Kevin Donovan (-> Brennan), after seeing Michael Caine movie about the Zulu uprising, decided to form the Universal Zulu Nation, an organization based on merits derived from art and achievements rather than violence and muscle. Gold labels: performer, adult, celebrity, actor, person Pred labels: actor, person, artist, director -> person Input: Most car spoilers are made from polyurethane, while some are made from lightweight steel or fiberglass. Gold labels: part, object Pred labels: …

Content truncated to 500 words