Probing Pre-trained Auto-regressive Language Models for Named Entity Typing and Recognition

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 1408–1417 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 1408 Probing Pre-trained Auto-regressive Language Models for Named Entity Typing and Recognition Elena V. Epure, Romain Hennequin Deezer Research 22-26 rue de Calais, 75009 Paris, France research@deezer.com Abstract Multiple works have proposed to probe language models (LMs) for generalization in named entity (NE) typing (NET) and recognition (NER). However, little has been done in this direction for auto-regressive models despite their popularity and potential to express a wide variety of NLP tasks in the same unified format. We propose a new methodology to probe auto-regressive LMs for NET and NER generalization, which draws inspiration from human linguistic behavior, by resorting to meta-learning. We study NEs of various types individually by designing a zero-shot transfer strategy for NET. Then, we probe the model for NER by providing a few examples at in- ference. We introduce a novel procedure to assess the model’s memorization of NEs and report the memorization’s impact on the results. Our findings show that: 1) GPT2, a common pre-trained auto-regressive LM, without any fine-tuning for NET or NER, performs the tasks fairly well; 2) name irregularity when common for a NE type could be an effective exploitable cue; 3) the model seems to rely more on NE than contextual cues in few-shot NER; 4) NEs with words absent during LM pre-training are very challenging for both NET and NER. Keywords: auto-regressive language model, probing, zero-shot NET, few-shot NER, memorization testing 1. Introduction Before transformer LMs (Devlin et al., 2019), the state- of-the-art NER was based on training recurrent neural net- works, such as bidirectional LSTM (BiLSTM) with a Con- ditional Random Field (CRF) layer, from scratch (Yadav and Bethard, 2018). The widely adopted approach with transformers has been to fine-tune them for the desired task, thus specializing their general linguistic knowledge, ac- quired during pre-training. While LMs fine-tuned for NER have achieved impressive results on standard benchmarks (Akbik et al., 2019), multiple works have emphasized their limitations with regard to their generalization capacity to new textual genres (e.g. clean versus noisy text), NE type sets (e.g. NE types belonging to new domains such as mu- sic or e-commerce) and new NEs, unseen during training (Lin et al., 2020b). To gain more insights into the NER generalization ability of LMs, multiple studies have been conducted. Probing has been designed for BiLSTM-CRF LMs (Augenstein et al., 2017; Taill´e et al., 2020; Fu et al., 2020) or masked LMs such as BERT (Petroni et al., 2019; Jiang et al., 2020). Yet, little has been done for auto-regressive models such as GPT2 despite their popularity and potential to express a wide variety of NLP tasks in the same unified format (Raf- fel et al., 2020). Additionally, although the past probing studies have broad- ened the knowledge about how LMs generalize in the NER context, multiple improvements could be brought to exist- ing methodologies. First, the …

Content truncated to 500 words