Capturing Long-distance Dependencies in Sequence Models: A Case Study of Chinese Part-of-speech Tagging

International Joint Conference on Natural Language Processing, pages 180–188, Nagoya, Japan, 14-18 October 2013. Capturing Long-distance Dependencies in Sequence Models: A Case Study of Chinese Part-of-speech Tagging Weiwei Sun and Xiaochang Peng and Xiaojun Wan Institute of Computer Science and Technology, Peking University The MOE Key Laboratory of Computational Linguistics {ws,wanxiaojun}@pku.edu.cn; pxc.pku@gmail.com Abstract This paper is concerned with capturing long-distance dependencies in sequence models. We propose a two-step strat- egy. First, the stacked learning technique is applied to integrate sequence models that are good at exploring local informa- tion and other high complexity models that are good at capturing long-distance de- pendencies. Second, the structure com- pilation technique is employed to trans- fer the predictive power of hybrid models to sequence models via large-scale unla- beled data. To investigate the feasibility of our idea, we study Chinese POS tag- ging. Experiments on the Chinese Tree- bank data demonstrate the effectiveness of our methods. The re-compiled models not only achieve high accuracy with respect to per token classification, but also serve as a front-end to a parser well. 1 Introduction Sequential classification models provide very im- portant solutions to pattern recognition tasks that involve the automatic assignment of a categorical label to each token of a sequence of observed val- ues. A common example is part-of-speech (POS) tagging, which seeks to assign a grammatical cat- egory to each word in an input sentence. Standard machine learning algorithms to sequential tagging, e.g. linear-chain conditional random fields and max-margin Markov network, directly exploit lo- cal dependencies and perform quite well for a large number of sequence labeling tasks. In these models, usually, the relationships between two (or three) successive labels are parameterized and en- coded as a single feature, and Viterbi style dy- namic programming algorithms are applied to in- ference over a lattice. Although sequence models perform well for many applications, they are inad- equate for tasks where many long-distance depen- dencies are involved. Sequential classification models play an impor- tant role in natural language processing (NLP). Several fundamental NLP tasks, including named entity recognition, POS tagging, text chunking, supertagging, etc., employ sequential classifiers for lexical and syntactic disambiguation. In ad- dition to learning linear chain structures, sequence models can even be applied to acquire hierarchical syntactic structures (Tsuruoka et al., 2009). How- ever, long-distance dependencies widely exist in linguistic structures, and many NLP systems suf- fer from the incapability of capturing these depen- dencies. For example, previous work has shown that sequence models alone cannot deal with syn- tactic ambiguities well (Clark and Curran, 2004; Tsuruoka et al., 2009). On the contrary, state- of-the-art systems usually utilize high complexity models, such as lexicalized PCFG models for syn- tactic parsing, to achieve high accuracy. Unfortu- nately, they are not suitable for many real world applications due to the sacrifice of efficiency. In this paper, we are concerned with capturing long-distance dependencies in sequence models. Our goal is to develop efficient models with lin- ear time complexity that are also capable to cap- …

Content truncated to 500 words