When and how to paraphrase for named entity recognition?

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 7052–7087 July 9-14, 2023 ©2023 Association for Computational Linguistics When and how to paraphrase for named entity recognition? Saket Sharma1¶, Aviral Joshi1¶, Yiyun Zhao1¶, Namrata Mukhija1¶, Hanoz Bhathena¶, Prateek Singh¶, Sashank Santhanam∥2 ¶Machine Learning Center of Excellence, JPMorgan Chase & Co. ∥Apple saket.sharma@jpmchase.com Abstract While paraphrasing is a promising approach for data augmentation in classification tasks, its effect on named entity recognition (NER) is not investigated systematically due to the difficulty of span-level label preservation. In this paper, we utilize simple strategies to annotate entity spans in generations and compare established and novel methods of paraphrasing in NLP such as back translation, specialized encoder- decoder models such as Pegasus, and GPT-3 variants for their effectiveness in improving downstream performance for NER across dif- ferent levels of gold annotations and paraphras- ing strength on 5 datasets. We thoroughly ex- plore the influence of paraphrasers, dynamics between paraphrasing strength and gold dataset size on the NER performance with visualiza- tions and statistical testing. We find that the choice of the paraphraser greatly impacts NER performance, with one of the larger GPT-3 vari- ants exceedingly capable of generating high quality paraphrases, yielding statistically signif- icant improvements in NER performance with increasing paraphrasing strength, while other paraphrasers show more mixed results. Addi- tionally, inline auto annotations generated by larger GPT-3 are strictly better than heuristic based annotations. We also find diminishing benefits of paraphrasing as gold annotations in- crease for most datasets. Furthermore, while most paraphrasers promote entity memoriza- tion in NER, the proposed GPT-3 configuration performs most favorably among the compared paraphrasers when tested on unseen entities, with memorization reducing further with para- phrasing strength. Finally, we explore men- tion replacement using GPT-3, which provides additional benefits over base paraphrasing for specific datasets. 1Equal contribution. 2Currently at Apple. Work done while at JPMorgan Chase & Co. 1 Introduction Named entity recognition (NER) seeks to extract entity mentions (e.g., Shakespeare, Warwickshire) from a text (Shakespeare was born and raised in Warwickshire) for predefined categories of inter- est (such as people and locations). It is a critical component underpinning many industrial pipelines for a variety of downstream natural language pro- cessing applications such as search, recommenda- tion, and virtual assistant systems. However, in real-world applications, there is often a scarcity of labeled data for training advanced deep neural mod- els because span-level NER annotations are costly, and domain expertise may be needed to annotate data from domains such as finance, biomedical sci- ences, etc. Data augmentation is often used as an alterna- tive to address the data scarcity issue in many NLP tasks (see an NLP data augmentation survey by Feng et al. (2021)). However, data augmentation for NER imposes additional challenges because NER requires token/span level label preservation. Therefore, most existing works on NER data aug- mentation primarily focus on local replacement for entity mentions (Dai and Adel, 2020; Zhou et al., 2022; Liu et al., 2022; …

Content truncated to 500 words