Embedded-State Latent Conditional Random Fields for Sequence Labeling

Proceedings of the 22nd Conference on Computational Natural Language Learning (CoNLL 2018), pages 1–10 Brussels, Belgium, October 31 - November 1, 2018. c⃝2018 Association for Computational Linguistics 1 Embedded-State Latent Conditional Random Fields for Sequence Labeling Dung Thai Sree Harsha Ramesh Shikhar Murty Luke Vilnis Andrew McCallum College of Information and Computer Sciences University of Massachusetts Amherst {dthai, shramesh, smurty, luke, mccallum}@cs.umass.edu Abstract Complex textual information extraction tasks are often posed as sequence labeling or shal- low parsing, where fields are extracted using local labels made consistent through proba- bilistic inference in a graphical model with constrained transitions. Recently, it has be- come common to locally parametrize these models using rich features extracted by recur- rent neural networks (such as LSTM), while enforcing consistent outputs through a sim- ple linear-chain model, representing Marko- vian dependencies between successive labels. However, the simple graphical model struc- ture belies the often complex non-local con- straints between output labels. For example, many fields, such as a first name, can only oc- cur a fixed number of times, or in the pres- ence of other fields. While RNNs have pro- vided increasingly powerful context-aware lo- cal features for sequence tagging, they have yet to be integrated with a global graphical model of similar expressivity in the output dis- tribution. Our model goes beyond the lin- ear chain CRF to incorporate multiple hidden states per output label, but parametrizes their transitions parsimoniously with low-rank log- potential scoring matrices, effectively learning an embedding space for hidden states. This augmented latent space of inference variables complements the rich feature representation of the RNN, and allows exact global infer- ence obeying complex, learned non-local out- put constraints. We experiment with several datasets and show that the model outperforms baseline CRF+RNN models when global out- put constraints are necessary at inference-time, and explore the interpretable latent structure. 1 Introduction As with many other prediction tasks involving complex structured outputs, such as image seg- mentation (Chen et al., 2018), machine transla- tion (Bahdanau et al., 2015), and speech recogni- tion (Hinton et al., 2012), deep neural networks (DNNs) for sequence labeling and shallow pars- ing have become standard tools for for information extraction (Collobert et al., 2011; Lample et al., 2016). In the language of structured prediction, DNNs process the input sequence to produce a rich local parametrization for the output prediction model. However, output variables obey a variety of hard and soft constraints — for example, in se- quence tagging tasks such as named entity recog- nition, I-PER cannot follow B-ORG. Interestingly, even with such powerful local fea- turization, the DNN model does not automatically capture a mode of the output distribution through local decisions alone, and can violate these con- straints. Successful applications of DNNs to se- quence tagging gain from incorporating a sim- ple linear chain probabilistic graphical model to enforce consistent output predictions (Collobert et al., 2011; Lample et al., 2016), and more gen- erally the addition of a graphical model to en- force output label consistency …

Content truncated to 500 words