SVM Model Tampering and Anchored Learning: A Case Study in Hebrew NP Chunking

Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics, pages 224–231, Prague, Czech Republic, June 2007. c⃝2007 Association for Computational Linguistics SVM Model Tampering and Anchored Learning: A Case Study in Hebrew NP Chunking Yoav Goldberg and Michael Elhadad Computer Science Department Ben Gurion University of the Negev P.O.B 653 Be’er Sheva 84105, Israel yoavg,elhadad@cs.bgu.ac.il Abstract We study the issue of porting a known NLP method to a language with little existing NLP resources, specifically Hebrew SVM-based chunking. We introduce two SVM-based methods – Model Tampering and Anchored Learning. These allow fine grained analysis of the learned SVM models, which provides guidance to identify errors in the training cor- pus, distinguish the role and interaction of lexical features and eventually construct a model with ∼10% error reduction. The re- sulting chunker is shown to be robust in the presence of noise in the training corpus, relies on less lexical features than was previously understood and achieves an F-measure perfor- mance of 92.2 on automatically PoS-tagged text. The SVM analysis methods also provide general insight on SVM-based chunking. 1 Introduction While high-quality NLP corpora and tools are avail- able in English, such resources are difficult to obtain in most other languages. Three challenges must be met when adapting results established in English to another language: (1) acquiring high quality anno- tated data; (2) adapting the English task definition to the nature of a different language, and (3) adapt- ing the algorithm to the new language. This paper presents a case study in the adaptation of a well known task to a language with few NLP resources available. Specifically, we deal with SVM based He- brew NP chunking. In (Goldberg et al., 2006), we established that the task is not trivially transferable to Hebrew, but reported that SVM based chunking (Kudo and Matsumoto, 2000) performs well. We extend that work and study the problem from 3 an- gles: (1) how to deal with a corpus that is smaller and with a higher level of noise than is available in English; we propose techniques that help identify ‘suspicious’ data points in the corpus, and identify how robust the model is in the presence of noise; (2) we compare the task definition in English and in Hebrew through quantitative evaluation of the differ- ences between the two languages by analyzing the relative importance of features in the learned SVM models; and (3) we analyze the structure of learned SVM models to better understand the characteristics of the chunking problem in Hebrew. While most work on chunking with machine learning techniques tend to treat the classification engine as a black-box, we try to investigate the re- sulting classification model in order to understand its inner working, strengths and weaknesses. We in- troduce two SVM-based methods – Model Tamper- ing and Anchored Learning – and demonstrate how a fine-grained analysis of SVM models provides in- sights on all three accounts. The understanding of the relative contribution of each feature in the model helps …

Content truncated to 500 words