BERTifying the Hidden Markov Model for Multi-Source Weakly Supervised Named Entity Recognition

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 6178–6190 August 1–6, 2021. ©2021 Association for Computational Linguistics 6178 BERTifying the Hidden Markov Model for Multi-Source Weakly Supervised Named Entity Recognition Yinghao Li1, Pranav Shetty1, Lucas Liu1, Chao Zhang1, and Le Song2 1 Georgia Institute of Technology, Atlanta, USA 2 Mohamed bin Zayed University of Artificial Intelligence, Abu Dhabi, United Arab Emirates {yinghaoli, pranav.shetty, lucasliu, chaozhang}@gatech.edu le.song@mbzuai.ac.ae Abstract We study the problem of learning a named en- tity recognition (NER) tagger using noisy la- bels from multiple weak supervision sources. Though cheap to obtain, the labels from weak supervision sources are often incomplete, in- accurate, and contradictory, making it difficult to learn an accurate NER model. To address this challenge, we propose a conditional hid- den Markov model (CHMM), which can effec- tively infer true labels from multi-source noisy labels in an unsupervised way. CHMM en- hances the classic hidden Markov model with the contextual representation power of pre- trained language models. Specifically, CHMM learns token-wise transition and emission prob- abilities from the BERT embeddings of the in- put tokens to infer the latent true labels from noisy observations. We further refine CHMM with an alternate-training approach (CHMM- ALT). It fine-tunes a BERT-NER model with the labels inferred by CHMM, and this BERT- NER’s output is regarded as an additional weak source to train the CHMM in return. Ex- periments on four NER benchmarks from var- ious domains show that our method outper- forms state-of-the-art weakly supervised NER models by wide margins. 1 Introduction Named entity recognition (NER), which aims to identify named entities from unstructured text, is an information extraction task fundamental to many downstream applications such as event detection (Li et al., 2012), relationship extraction (Bach and Badaskar, 2007), and question answering (Khalid et al., 2008). Existing NER models are typically supervised by a large number of training sequences, each pre-annotated with token-level labels. In prac- tice, however, obtaining such labels could be pro- hibitively expensive. On the other hand, many do- mains have various knowledge resources such as knowledge bases, domain-specific dictionaries, or labeling rules provided by domain experts (Far- makiotou et al., 2000; Nadeau and Sekine, 2007). These resources can be used to match a corpus and quickly create large-scale noisy training data for NER from multiple views. Learning an NER model from multiple weak su- pervision sources is a challenging problem. While there are works on distantly supervised NER that use only knowledge bases as weak supervision (Mintz et al., 2009; Shang et al., 2018; Cao et al., 2019; Liang et al., 2020), they cannot leverage complementary information from multiple annota- tion sources. To handle multi-source weak super- vision, several recent works (Nguyen et al., 2017; Safranchik et al., 2020; Lison et al., 2020) leverage the hidden Markov model (HMM), by modeling true labels as hidden variables and inferring them from the observed noisy labels through unsuper- vised learning. Though principled, …

Content truncated to 500 words