Chunking with Support Vector Machines

Chunking with Support Vector Machines Taku Kudo and Yuji Matsumoto Graduate School of Information Science, Nara Institute of Science and Technology � taku-ku,matsu � @is.aist-nara.ac.jp Abstract We apply Support Vector Machines (SVMs) to identify English base phrases (chunks). SVMs are known to achieve high generalization perfor- mance even with input data of high dimensional feature spaces. Furthermore, by the Kernel princi- ple, SVMs can carry out training with smaller com- putational overhead independent of their dimen- sionality. We apply weighted voting of 8 SVMs- based systems trained with distinct chunk repre- sentations. Experimental results show that our ap- proach achieves higher accuracy than previous ap- proaches. 1 Introduction Chunking is recognized as series of processes — first identifying proper chunks from a sequence of tokens (such as words), and second classifying these chunks into some grammatical classes. Various NLP tasks can be seen as a chunking task. Exam- ples include English base noun phrase identification (base NP chunking), English base phrase identifica- tion (chunking), Japanese chunk (bunsetsu) identi- fication and named entity extraction. Tokenization and part-of-speech tagging can also be regarded as a chunking task, if we assume each character as a token. Machine learning techniques are often applied to chunking, since the task is formulated as estimating an identifying function from the information (fea- tures) available in the surrounding context. Various machine learning approaches have been proposed for chunking (Ramshaw and Marcus, 1995; Tjong Kim Sang, 2000a; Tjong Kim Sang et al., 2000; Tjong Kim Sang, 2000b; Sassano and Utsuro, 2000; van Halteren, 2000). Conventional machine learning techniques, such as Hidden Markov Model (HMM) and Maximum Entropy Model (ME), normally require a careful feature selection in order to achieve high accuracy. They do not provide a method for automatic selec- tion of given feature sets. Usually, heuristics are used for selecting effective features and their com- binations. New statistical learning techniques such as Sup- port Vector Machines (SVMs) (Cortes and Vap- nik, 1995; Vapnik, 1998) and Boosting(Freund and Schapire, 1996) have been proposed. These tech- niques take a strategy that maximizes the margin between critical samples and the separating hyper- plane. In particular, SVMs achieve high generaliza- tion even with training data of a very high dimen- sion. Furthermore, by introducing the Kernel func- tion, SVMs handle non-linear feature spaces, and carry out the training considering combinations of more than one feature. In the field of natural language processing, SVMs are applied to text categorization and syntactic de- pendency structure analysis, and are reported to have achieved higher accuracy than previous ap- proaches.(Joachims, 1998; Taira and Haruno, 1999; Kudo and Matsumoto, 2000a). In this paper, we apply Support Vector Machines to the chunking task. In addition, in order to achieve higher accuracy, we apply weighted voting of 8 SVM-based systems which are trained using dis- tinct chunk representations. For the weighted vot- ing systems, we introduce a new type of weighting strategy which are derived from the theoretical basis of the SVMs. 2 2.1Support Vector Machines Optimal Hyperplane Let …

Content truncated to 500 words