Context-aware Adversarial Training for Name Regularity Bias in Named Entity Recognition

Context-aware Adversarial Training for Name Regularity Bias in Named Entity Recognition Abbas Ghaddar, Philippe Langlais†, Ahmad Rashid, and Mehdi Rezagholizadeh Huawei Noah’s Ark Lab, Montreal Research Center, Canada †RALI/DIRO, Universit´e de Montr´eal, Canada abbas.ghaddar@huawei.com, felipe@iro.umontreal.ca ahmad.rashid@huawei.com, mehdi.rezagholizadeh@huawei.com Abstract In this work, we examine the ability of NER models to use contextual information when predicting the type of an ambiguous entity. We introduce NRB, a new testbed carefully designed to diagnose Name Regularity Bias of NER models. Our results indicate that all state- of-the-art models we tested show such a bias; BERT fine-tuned models significantly outper- forming feature-based (LSTM-CRF) ones on NRB, despite having comparable (sometimes lower) performance on standard benchmarks. To mitigate this bias, we propose a novel model-agnostic training method that adds learn- able adversarial noise to some entity mentions, thus enforcing models to focus more strongly on the contextual signal, leading to significant gains on NRB. Combining it with two other training strategies, data augmentation and pa- rameter freezing, leads to further gains. 1 Introduction Recent advances in language model pre-training (Peters et al., 2018; Devlin et al., 2019; Liu et al., 2019) have greatly improved the performance of many Natural Language Understanding (NLU) tasks. Yet, several studies (McCoy et al., 2019; Clark et al., 2019; Utama et al., 2020b) revealed that state-of-the-art NLU models often make use of surface patterns in the data that do not gen- eralize well. Named-Entity Recognition (NER), a downstream task that consists in identifying textual mentions and classifying them into a predefined set of types, is no exception. The robustness of modern NER models has received considerable attention recently (Mayhew et al., 2019; Mayhew et al., 2020; Agarwal et al., 2020a; Zeng et al., 2020; Bernier-Colborne and Langlais, 2020). Name Regularity Bias (Lin et al., 2020; Agarwal et al., 2020b; Zeng et al., 2020) in NER occurs when a model relies on a signal coming from the entity name, and disregards evidence within the local context. Figure 1 shows examples where state-of-the-art models (Peters et al., 2018; Akbik et al., 2018; Devlin et al., 2019) fail to exploit contextual information. For instance, the entity Gonzales in the first sentence of the figure is wrongly recognized as a per- son, while the context clearly signals that it is a location (city). To better highlight this issue, we propose NRB, a testbed designed to accurately diagnose name regularity bias of NER models by harvesting natural sentences from Wikipedia that contain challenging entities, such as those in Figure 1. This is different from previous work that evalu- ated models on artificial data obtained by either randomizing (Lin et al., 2020) or substituting entities by ones from a pre-defined list (Agarwal et al., 2020a). NRB is compatible with any anno- tation scheme, and is intended to be used as an auxiliary validation set. We conduct experiments with the feature- based LSTM-CRF architecture (Peters et al., 2018; Akbik et al., 2018) and the BERT (Devlin et al., 2019) fine-tuning approach trained on stan- dard benchmarks. The best …

Content truncated to 500 words