Uncertainty-Aware Label Refinement for Sequence Labeling

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 2316–2326, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 2316 Uncertainty-Aware Label Refinement for Sequence Labeling Tao Gui∗12, Jiacheng Ye∗12, Qi Zhang12, Zhengyan Li12, Zichu Fei12, Yeyun Gong3 and Xuanjing Huang12 1Shanghai Key Laboratory of Intelligent Information Processing, Fudan University 2School of Computer Science, Fudan University 3Microsoft Research Asia {tgui16, yejc19, qz, lizy19, zcfei19, xjhuang}@fudan.edu.cn yegong@microsoft.com Abstract Conditional random fields (CRF) for label decoding has become ubiquitous in sequence labeling tasks. However, the local label dependencies and inefficient Viterbi decoding have always been a problem to be solved. In this work, we introduce a novel two-stage label decoding framework to model long-term label dependencies, while being much more computationally efficient. A base model first predicts draft labels, and then a novel two- stream self-attention model makes refinements on these draft predictions based on long- range label dependencies, which can achieve parallel decoding for a faster prediction. In addition, in order to mitigate the side effects of incorrect draft labels, Bayesian neural net- works are used to indicate the labels with a high probability of being wrong, which can greatly assist in preventing error propagation. The experimental results on three sequence labeling benchmarks demonstrated that the proposed method not only outperformed the CRF-based methods but also greatly acceler- ated the inference process. 1 Introduction Linguistic sequence labeling is one of the funda- mental tasks in natural language processing. It has the goal of predicting a linguistic label for each word, including part-of-speech (POS) tagging, text chunking, and named entity recognition (NER). Benefiting from representation learning, neural network-based approaches can achieve state-of- the-art performance without massive handcrafted feature engineering (Ma and Hovy, 2016; Lample et al., 2016; Strubell et al., 2017; Peters et al., 2018; Devlin et al., 2019). Although the use of representation learning to obtain better text representation is very successful, ∗Both authors contributed equally. United Arab Emirates B-LOC I-LOC E-LOC Input True Label Draft Label Refinement ... ... B-ORG I-LOC E-ORG B-LOC I-ORG E-LOC X <latexit sha1_base64="ex7UVx6LxzCcCSDqKZWFuZK2gBU=">ACzXicjVHLSsNAFD2N r1pfVZdugkVwVZIq6Loxp0V7APbIsl02obmxWQilKpbf8Ct/pb4B/oX3hlTUIvohCRnzr3nzNx73dj3EmlZrzljbn5hcSm/XFhZXVvfKG5uNZIoFYzXWeRHouU6Cfe9kNelJ3eigV3Atfn TXd0quLNGy4SLwov5Tjm3cAZhF7fY4k6qrDhpyNAkeMroslq2zpZc4COwMlZKsWFV/Q8RGFIE4AghCftwkNDThg0LMXFdTIgThDwd57hDgbQpZXHKcIgd0XdAu3bGhrRXnolWMzrFp1eQ 0sQeaSLKE4TVaOp9pZsb95T7SnutuY/m7mFRArMST2L908786VYtEH8e6Bo9qijWjqmOZS6q7om5ufqlKkNMnMI9igvCTCunfTa1JtG1q946Ov6mMxWr9izLTfGubkDtn+OcxY0KmX 7oFy5OCxVT7JR57GDXezTPI9QxRlqJN3iEc84dk4N1Lj1rj/TDVymWYb35bx8AFBiJNo</latexit> ⇥ <latexit sha1_base64="PcNBZ+k3dnNGojhYt75BSmjl47k=">ACyXicjVHLSsNAFD2 Nr1pfVZdugkVwVZIq6LoRnBTwT6gLZJMp3VsXiYTsRZX/oBb/THxD/QvDOmoBbRCUnOnHvPmbn3upEnEmlZrzljZnZufiG/WFhaXldK65vNJIwjRmvs9AL45brJNwTAa9LIT3eimLu +K7Hm+7wWMWbNzxORBicy1HEu74zCERfMEcS1ehI4fPkoliypZe5jSwM1BCtmph8QUd9BCIYUPjgCSsAcHCT1t2LAQEdfFmLiYkNBxjnsUSJtSFqcMh9ghfQe0a2dsQHvlmWg1o1M8e mNSmtghTUh5MWF1mqnjqXZW7G/eY+2p7jaiv5t5+cRKXBL7l26S+V+dqkWij0Ndg6CaIs2o6ljmkuquqJubX6qS5BARp3CP4jFhpWTPptak+jaVW8dHX/TmYpVe5blpnhXt6QB2z/HOQ 0albK9V6c7ZeqR9mo89jCNnZpngeo4gQ1Mn7Co94wrNxalwbt8bdZ6qRyzSb+LaMhw/935G5</latexit> ⇥ <latexit sha1_base64="PcNBZ+k3dnNGojhYt75BSmjl47k=">ACyXicjVHLSsNAFD2 Nr1pfVZdugkVwVZIq6LoRnBTwT6gLZJMp3VsXiYTsRZX/oBb/THxD/QvDOmoBbRCUnOnHvPmbn3upEnEmlZrzljZnZufiG/WFhaXldK65vNJIwjRmvs9AL45brJNwTAa9LIT3eimLu +K7Hm+7wWMWbNzxORBicy1HEu74zCERfMEcS1ehI4fPkoliypZe5jSwM1BCtmph8QUd9BCIYUPjgCSsAcHCT1t2LAQEdfFmLiYkNBxjnsUSJtSFqcMh9ghfQe0a2dsQHvlmWg1o1M8e mNSmtghTUh5MWF1mqnjqXZW7G/eY+2p7jaiv5t5+cRKXBL7l26S+V+dqkWij0Ndg6CaIs2o6ljmkuquqJubX6qS5BARp3CP4jFhpWTPptak+jaVW8dHX/TmYpVe5blpnhXt6QB2z/HOQ 0albK9V6c7ZeqR9mo89jCNnZpngeo4gQ1Mn7Co94wrNxalwbt8bdZ6qRyzSb+LaMhw/935G5</latexit> ⇥ <latexit sha1_base64="PcNBZ+k3dnNGojhYt75BSmjl47k=">ACyXicjVHLSsNAFD2 Nr1pfVZdugkVwVZIq6LoRnBTwT6gLZJMp3VsXiYTsRZX/oBb/THxD/QvDOmoBbRCUnOnHvPmbn3upEnEmlZrzljZnZufiG/WFhaXldK65vNJIwjRmvs9AL45brJNwTAa9LIT3eimLu +K7Hm+7wWMWbNzxORBicy1HEu74zCERfMEcS1ehI4fPkoliypZe5jSwM1BCtmph8QUd9BCIYUPjgCSsAcHCT1t2LAQEdfFmLiYkNBxjnsUSJtSFqcMh9ghfQe0a2dsQHvlmWg1o1M8e mNSmtghTUh5MWF1mqnjqXZW7G/eY+2p7jaiv5t5+cRKXBL7l26S+V+dqkWij0Ndg6CaIs2o6ljmkuquqJubX6qS5BARp3CP4jFhpWTPptak+jaVW8dHX/TmYpVe5blpnhXt6QB2z/HOQ 0albK9V6c7ZeqR9mo89jCNnZpngeo4gQ1Mn7Co94wrNxalwbt8bdZ6qRyzSb+LaMhw/935G5</latexit> Figure 1: Schematic of label refinement framework (Cui and Zhang, 2019). The goal is refining the label of “Arab” using contextual labels and words, while the refinement of other correct labels may be negatively impacted by incorrect draft labels. creating better models for label dependencies has always been the focus of sequence labeling tasks (Collobert et al., 2011; Ye and Ling, 2018; Zhang et al., 2018). Among them, the CRF layer integrated with neural encoders to capture label transition patterns (Zhou and Xu, 2015; Ma and Hovy, 2016) has become ubiquitous in sequence labeling tasks. However, CRF only captures the neighboring label dependencies and must rely on inefficient Viterbi decoding. Many of the recent methods try to introduce label embeddings to manage longer ranges of dependencies, such as two-stage label refinement (Krishnan and Manning, …

Content truncated to 500 words