Regularized Structured Perceptron: A Case Study on Chinese Word Segmentation, POS Tagging and Parsing

Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics, pages 164–173, Gothenburg, Sweden, April 26-30 2014. c⃝2014 Association for Computational Linguistics Regularized Structured Perceptron: A Case Study on Chinese Word Segmentation, POS Tagging and Parsing Kaixu Zhang Xiamen University Fujian, P.R. China kareyzhang@gmail.com Jinsong Su Xiamen University Fujian, P.R. China jssu@xmu.edu.cn Changle Zhou Xiamen University Fujian, P.R. China dozero@xmu.edu.cn Abstract Structured perceptron becomes popular for various NLP tasks such as tagging and parsing. Practical studies on NLP did not pay much attention to its regularization. In this paper, we study three simple but effec- tive task-independent regularization meth- ods: (1) one is to average weights of dif- ferent trained models to reduce the bias caused by the specific order of the train- ing examples; (2) one is to add penalty term to the loss function; (3) and one is to randomly corrupt the data flow during training which is called dropout in the neu- ral network. Experiments are conducted on three NLP tasks, namely Chinese word segmentation, part-of-speech tagging and dependency parsing. Applying proper reg- ularization methods or their combinations, the error reductions with respect to the av- eraged perceptron for some of these tasks can be up to 10%. 1 Introduction Structured perceptron is a linear classification al- gorithm. It is used for word segmentation (Zhang and Clark, 2011), POS (part-of-speech) tagging (Collins, 2002), syntactical parsing (Collins and Roark, 2004), semantical parsing (Zettlemoyer and Collins, 2009) and other NLP tasks. The averaged perceptron or the voted percep- tron (Collins, 2002) is proposed for better gener- alization. Early update (Collins and Roark, 2004; Huang et al., 2012) is used for inexact decod- ing algorithms such as the beam search. Dis- tributed training (McDonald et al., 2010) and the minibatch and parallelization method (Zhao and Huang, 2013) are recently proposed. Some other related work focuses on the task-specified feature engineering. Regularization is to improve the ability of generalization and avoid over-fitting for machine learning algorithms including online learning al- gorithms (Do et al., 2009; Xiao, 2010). But prac- tical studies on NLP did not pay much attention to the regularization of the structured perceptron. As a result, for some tasks the model learned using perceptron algorithm is not as good as the model learned using regularized condition random field. In this paper, we treat the perceptron algorithm as a special case of the stochastic gradient de- scent (SGD) algorithm and study three kinds of simple but effective task-independent regulariza- tion methods that can be applied. The averaging method is to average the weight vectors of differ- ent models. We propose a “shuffle-and-average” method to reduce the bias caused by the specific order of the training examples. The traditional penalty method is to add penalty term to the loss function. The dropout method is to randomly cor- rupt the data flow during training. We show that this dropout method originally used in neural net- work also helps the structured perceptron. In Section 2, we describe the …

Content truncated to 500 words