Masked Conditional Random Fields for Sequence Labeling

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2024–2035 June 6–11, 2021. ©2021 Association for Computational Linguistics 2024 Masked Conditional Random Fields for Sequence Labeling Tianwen Wei∗ Jianwei Qi Shenghuan He Songtao Sun Xiaomi AI {weitianwen,qijianwei,heshenghuan,sunsongtao}@xiaomi.com Abstract Conditional Random Field (CRF) based neural models are among the most performant meth- ods for solving sequence labeling problems. Despite its great success, CRF has the short- coming of occasionally generating illegal se- quences of tags, e.g. sequences containing an “I-” tag immediately after an “O” tag, which is forbidden by the underlying BIO tagging scheme. In this work, we propose Masked Conditional Random Field (MCRF), an easy to implement variant of CRF that impose re- strictions on candidate paths during both train- ing and decoding phases. We show that the proposed method thoroughly resolves this is- sue and brings consistent improvement over existing CRF-based models with near zero ad- ditional cost. 1 Introduction Sequence labeling problems such as named entity recognition (NER), part of speech (POS) tagging and chunking have long been considered as funda- mental NLP tasks and drawn researcher’s attention for many years. Traditional work is based on statistical ap- proaches such as Hidden Markov Models (Baum and Petrie, 1966) and Conditional Random Fields (Lafferty et al., 2001), where handcrafted features and task-specific resources are used. With advances in deep learning, neural network based models have achieved dominance in sequence labeling tasks in an end-to-end manner. Those models typically con- sist of a neural encoder that maps the input tokens to embeddings capturing global sequence informa- tion, and a CRF layer that models dependencies be- tween neighboring labels. Popular choices of neu- ral encoder have been convolutional neural network (Collobert et al., 2011), and bidirectional LSTM (Huang et al., 2015). Recently, pretrained language models such as ELMo (Peters et al., 2018) or BERT ∗ Corresponding author. (Devlin et al., 2019) have been proven far superior as a sequence encoder, achieving state-of-the-art results on a broad range of sequence labeling tasks. Most sequence labeling models adopt a BIO or BIOES tag encoding scheme (Ratinov and Roth, 2009), which forbids certain tag transitions by de- sign. Occasionally, a model may yield sequence of predicted tags that violates the rules of the scheme. Such predictions, subsequently referred to as illegal paths, are erroneous and must be dealt with. Exist- ing methods rely on hand-crafted post-processing procedure to resolve this problem, typically by re- taining the illegal segments and re-tagging them. But as we shall show in this work, such treatment is arbitrary and leads to suboptimal performance. The main contribution of this paper is to give a principled solution to the illegal path problem. More precisely: 1. We show that in the neural-CRF framework the illegal path problem is intrinsic and may accounts for non-negligible proportion (up to 40%) of total errors. To the best of our knowl- edge we are the first to conduct this kind of study. 2. We …

Content truncated to 500 words