MELM: Data Augmentation with Masked Entity Language Modeling for Low-Resource NER

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 2251 - 2262 May 22-27, 2022 c⃝2022 Association for Computational Linguistics MELM: Data Augmentation with Masked Entity Language Modeling for Low-Resource NER Ran Zhou∗1,2 Xin Li†1 Ruidan He1 Lidong Bing1 Erik Cambria2 Luo Si1 Chunyan Miao2 1DAMO Academy, Alibaba Group 2Nanyang Technological University, Singapore {ran.zhou, xinting.lx, ruidan.he, l.bing, luo.si}@alibaba-inc.com {cambria, ascymiao}@ntu.edu.sg Abstract Data augmentation is an effective solution to data scarcity in low-resource scenarios. How- ever, when applied to token-level tasks such as NER, data augmentation methods often suf- fer from token-label misalignment, which leads to unsatsifactory performance. In this work, we propose Masked Entity Language Modeling (MELM) as a novel data augmentation frame- work for low-resource NER. To alleviate the token-label misalignment issue, we explicitly inject NER labels into sentence context, and thus the fine-tuned MELM is able to predict masked entity tokens by explicitly condition- ing on their labels. Thereby, MELM gener- ates high-quality augmented data with novel entities, which provides rich entity regular- ity knowledge and boosts NER performance. When training data from multiple languages are available, we also integrate MELM with code- mixing for further improvement. We demon- strate the effectiveness of MELM on mono- lingual, cross-lingual and multilingual NER across various low-resource levels. Experimen- tal results show that our MELM presents sub- stantial improvement over the baseline meth- ods.1 1 Introduction Named entity recognition (NER) is a fundamen- tal NLP task which aims to locate named entity mentions and classify them into predefined cat- egories. As a subtask of information extraction, it serves as a key building block for information retrieval (Banerjee et al., 2019), question answer- ing (Fabbri et al., 2020) and text summarization sys- tems (Nallapati et al., 2016) etc. However, except a few high-resource languages / domains, the ma- jority of languages / domains have limited amount ∗ Ran Zhou is under the Joint Ph.D. Program between Alibaba and Nanyang Technological University. †Corresponding author 1Our code is available at https://github.com/ RandyZhouRan/MELM/. of labeled data. Since manually annotating sufficient labeled data for each language / domain is expensive, low- resource NER (Cotterell and Duh, 2017; Feng et al., 2018; Zhou et al., 2019; Rijhwani et al., 2020) has received increasing attention in the research com- munity over the past years. As an effective solu- tion to data scarcity in low-resource scenarios, data augmentation enlarges the training set by apply- ing label-preserving transformations. Typical data augmentation methods for NLP include (1) word- level modification (Wei and Zou, 2019; Kobayashi, 2018; Wu et al., 2019; Kumar et al., 2020) and (2) back-translation (Sennrich et al., 2016; Fadaee et al., 2017; Dong et al., 2017; Yu et al., 2018). Despite the effectiveness on sentence-level tasks, they suffer from the token-label misalignment issue when applied to token-level tasks like NER. More specifically, word-level modification might replace an entity with alternatives that mismatch the origi- nal label. Back-translation creates augmented texts that largely preserve the original content. How- ever, it hinges …

Content truncated to 500 words