Segment-Level Sequence Modeling using Gated Recursive Semi-Markov Conditional Random Fields

Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pages 1413–1423, Berlin, Germany, August 7-12, 2016. c⃝2016 Association for Computational Linguistics Segment-Level Sequence Modeling using Gated Recursive Semi-Markov Conditional Random Fields Jingwei Zhuo1,2 ∗, Yong Cao2, Jun Zhu1 †, Bo Zhang1, Zaiqing Nie2 1Dept. of Comp. Sci. & Tech., State Key Lab of Intell. Tech. & Sys., TNList Lab, Tsinghua University, Beijing, 100084, China 2Microsoft Research, Beijing, 100084, China {zjw15@mails, dcszj@mail, dcszb@mail}.tsinghua.edu.cn; {yongc, znie}@microsoft.com Abstract Most of the sequence tagging tasks in nat- ural language processing require to recog- nize segments with certain syntactic role or semantic meaning in a sentence. They are usually tackled with Conditional Ran- dom Fields (CRFs), which do indirect word-level modeling over word-level fea- tures and thus cannot make full use of segment-level information. Semi-Markov Conditional Random Fields (Semi-CRFs) model segments directly but extracting segment-level features for Semi-CRFs is still a very challenging problem. This pa- per presents Gated Recursive Semi-CRFs (grSemi-CRFs), which model segments directly and automatically learn segment- level features through a gated recursive convolutional neural network. Our exper- iments on text chunking and named en- tity recognition (NER) demonstrate that grSemi-CRFs generally outperform other neural models. 1 Introduction Most of the sequence tagging tasks in natural lan- guage processing (NLP) are segment-level tasks, such as text chunking and named entity recog- nition (NER), which require to recognize seg- ments (i.e., a set of continuous words) with cer- tain syntactic role or semantic meaning in a sen- tence. These tasks are usually tackled with Con- ditional Random Fields (CRFs) (Lafferty et al., 2001), which do word-level modeling as putting each word a tag, by using some predefined tag- ging schemes, e.g., the “IOB” scheme (Ramshaw ∗ This work was done when J.W.Z was on an internship with Microsoft Research. † J.Z is the corresponding author. and Marcus, 1995). Such tagging schemes are lossy transformations of original segment tags: They do indicate the boundary of adjacent seg- ments but lose the length information of segments to some extent. Besides, CRFs can only employ word-level features, which are either hand-crafted or extracted with deep neural networks, such as window-based neural networks (Collobert et al., 2011) and bidirectional Long Short-Term Mem- ory networks (BI-LSTMs) (Huang et al., 2015). Therefore, CRFs cannot make full use of segment- level information, such as inner properties of seg- ments, which cannot be fully encoded in word- level features. Semi-Markov Conditional Random Fields (Semi-CRFs) (Sarawagi and Cohen, 2004) are proposed to model segments directly and thus readily utilize segment-level features that encode useful segment information. Existing work has shown that Semi-CRFs outperform CRFs on segment-level tagging tasks such as sequence segmentation (Andrew, 2006), NER (Sarawagi and Cohen, 2004; Okanohara et al., 2006), web data extraction (Zhu et al., 2007) and opinion extraction (Yang and Cardie, 2012). However, Semi-CRFs need many more features compared to CRFs as they need to model segments with different lengths. As manually designing the features is tedious and often incomplete, how to automatically extract …

Content truncated to 500 words