Training for Gibbs Sampling on Conditional Random Fields with Neural Scoring Factors

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 4999–5011, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 4999 Training for Gibbs Sampling on Conditional Random Fields with Neural Scoring Factors Sida Gao Carnegie Mellon University sidag@alumni.cmu.edu Matthew R. Gormley Carnegie Mellon University mgormley@cs.cmu.edu Abstract Most recent improvements in NLP come from changes to the neural network architectures modeling the text input. Yet, state-of-the- art models often rely on simple approaches to model the label space, e.g. bigram Con- ditional Random Fields (CRFs) in sequence tagging. More expressive graphical models are rarely used due to their prohibitive com- putational cost. In this work, we present an approach for efficiently training and decod- ing hybrids of graphical models and neural networks based on Gibbs sampling. Our ap- proach is the natural adaptation of SampleR- ank (Wick et al., 2011) to neural models, and is widely applicable to tasks beyond sequence tagging. We apply our approach to named entity recognition and present a neural skip- chain CRF model, for which exact inference is impractical. The skip-chain model improves over a strong baseline on three languages from CoNLL-02/03. We obtain new state-of-the-art results on Dutch. 1 1 Introduction Complex probabilistic graphical models were widely adopted for NLP tasks before the prevalence of deep learning (e.g. the skip-chain CRF of Finkel et al. (2005) and Sutton and Mccallum (2004) for NER). Although modern neural architectures are able to learn much better feature representations (e.g. the contextualized word representations of Peters et al. (2018), Devlin et al. (2018), and Ak- bik et al. (2019)) than the hand-crafted features used classically in graphical model’s log-linear po- tentials, these advances in feature learning do not negate the need for modeling the output label space. Consider two contrasting approaches to struc- tured prediction: transition-based models and 1The code is available at https://github.com/ GaoSida/Neural-SampleRank. graphical models. Transition-based models (e.g. the sequence-to-sequence models of Sutskever et al. (2014)) have enjoyed recent success thanks to their ability to have unbounded memory of past tran- sitions when predicting subsequent ones; yet be- cause no conditional independence assumptions are made, inference is typically restricted to (heuris- tic) greedy search and its variants. By contrast, graphical models make strong conditional indepen- dence assumptions, but enjoy a wealth of inference algorithms, both exact and approximate, as a re- sult. Moreover, graphical models readily admit the incorporation of domain knowledge about interac- tions between the output variables. In this paper, we focus on this latter approach to modeling. Specifically, we explore conditional random fields (CRFs) (Lafferty et al., 2001) with neural po- tential functions. Prior state-of-the-art approaches utilizing such models (e.g. CRF-LSTMs) for se- quence tagging tasks like named entity recognition (NER) have focused on simple linear-chain CRFs, which only model bi-gram dependencies of adja- cent labels (Lample et al., 2016; Peters et al., 2017), and the exact inference can be done in polynomial time with dynamic programming. By contrast, we are motivated by CRFs that do not admit exact inference. We …

Content truncated to 500 words