A Rigorous Study on Named Entity Recognition: Can Fine-tuning Pretrained Model Lead to the Promised Land?

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 7291–7300, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 7291 A Rigorous Study on Named Entity Recognition: Can Fine-tuning Pretrained Model Lead to the Promised Land? Hongyu Lin1, Yaojie Lu1,3, Jialong Tang1,3, Xianpei Han1,2,˚ , Le Sun1,2,˚ Zhicheng Wei4 , Nicholas Jing Yuan4 1Chinese Information Processing Laboratory 2State Key Laboratory of Computer Science Institute of Software, Chinese Academy of Sciences, Beijing, China 3University of Chinese Academy of Sciences, Beijing, China 4Huawei Cloud&AI {hongyu,yaojie2017,jialong2019,xianpei,sunle}@iscas.ac.cn {weizhicheng1,nicholas.yuan}@huawei.com Abstract Fine-tuning pretrained model has achieved promising performance on standard NER benchmarks. Generally, these benchmarks are blessed with strong name regularity, high mention coverage and sufficient context di- versity. Unfortunately, when scaling NER to open situations, these advantages may no longer exist. And therefore it raises a crit- ical question of whether previous creditable approaches can still work well when facing these challenges. As there is no currently available dataset to investigate this problem, this paper proposes to conduct randomization test on standard benchmarks. Specifically, we erase name regularity, mention coverage and context diversity respectively from the bench- marks, in order to explore their impact on the generalization ability of models. To further verify our conclusions, we also construct a new open NER dataset that focuses on entity types with weaker name regularity and lower mention coverage to verify our conclusion. From both randomization test and empirical experiments, we draw the conclusions that 1) name regularity is critical for the models to generalize to unseen mentions; 2) high mention coverage may undermine the model generalization ability and 3) context patterns may not require enormous data to capture when using pretrained encoders. 1 Introduction Named entity recognition (NER), or more gen- erally name tagging, aims to identify text spans pertaining to specific entity types. NER is a fun- damental task of information extraction which en- ables many downstream NLP applications, such as relation extraction (GuoDong et al., 2005; Mintz et al., 2009), event extraction (Ji and Grishman, 2008; Li et al., 2013) and machine reading com- prehension (Rajpurkar et al., 2016; Wang et al., *: Corresponding authors. Regular NER Open NER Typical Categories Person, Location, Organization, etc. Movie, Song, Book, TV Series, etc. Name Regularity Entity types with strong regularity Entity types with weak or no regularity Mention Coverage Training set with high mention coverage Many new and unseen mentions Context Pattern With decent training instances to capture Fully-annotated training data is rare Examples … at [Cherry Street]… … go to [9th Avenue] … starting from [Cherry Street] … at [8th Avenue] … I watched [avatar]last night …[the matrix] is the best... Wow...[Joker] was great! Love [inception] so much. Location Movie Test Train Train Test Figure 1: Comparison between regular NER bench- marks and open NER tasks in reality. 2016). Recently, neural network-based supervised models dominate the NER task. By supervised fine-tuning upon large-scale language model pre- trained architectures (e.g., ELMo (Peters et al., 2018), BERT (Devlin et al., 2018), XLNet (Yang et al., …

Content truncated to 500 words