Detecting Errors in Corpora Using Support Vector Machines

Detecting Errors in Corpora Using Support Vector Machines Tetsuji Nakagawa∗ and Yuji Matsumoto Graduate School of Information Science Nara Institute of Science and Technology 8916−5 Takayama, Ikoma, Nara 630−0101, Japan nakagawa378@oki.com, matsu@is.aist-nara.ac.jp Abstract While the corpus-based research relies on hu- man annotated corpora, it is often said that a non-negligible amount of errors remain even in frequently used corpora such as Penn Treebank. Detection of errors in annotated corpora is im- portant for corpus-based natural language pro- cessing. In this paper, we propose a method to detect errors in corpora using support vec- tor machines (SVMs). This method is based on the idea of extracting exceptional elements that violate consistency. We propose a method of using SVMs to assign a weight to each ele- ment and to find errors in a POS tagged corpus. We apply the method to English and Japanese POS-tagged corpora and achieve high precision in detecting errors. 1 Introduction Corpora are widely used in natural language processing today. For example, many statistical part-of-speech (POS) taggers have been devel- oped and they use corpora as the training data to obtain statistical information or rules (Brill, 1995; Ratnaparkhi, 1996). For natural language processing systems based on a corpus, the quan- tity and quality of the corpus affect their per- formance. In general, corpora are annotated by hand, and therefore are error-prone. These errors are problematic for corpus-based sys- tems. The errors become false training exam- ples and deteriorate the performance of the sys- tems. Furthermore, incorrect instances may be used as testing examples and prevent the accu- rate measurement of performance. Many stud- ies and improvements have been conducted for ∗ Presently with Service Media Laboratory, Corporate Research and Development Center, Oki Electric Industry Co.,Ltd. POS tagging, and major methods of POS tag- ging achieve an accuracy of 96–97% on the Penn Treebank WSJ corpus, but obtaining higher ac- curacies is difficult (Ratnaparkhi, 1996). It is mentioned that the limitation is largely caused by inconsistencies in the corpus (Ratnaparkhi, 1996; Padr´o and M`arquez, 1998; van Halteren et al., 2001). Therefore, correcting the errors in a corpus and improving its quality is important. However, to find and correct errors in corpora by hand is costly, since the size of corpora is usually very large. Hence, automatic detection of errors in corpora is necessary. One of the approaches for corpus error de- tection is use of machine learning techniques (Abney et al., 1999; Matsumoto and Yamashita, 2000; Ma et al., 2001). These methods regard difficult elements for a learning model (boosting or neural networks) to learn as corpus errors. Abney et al. (1999) studied corpus error detec- tion using boosting. Boosting assigns weights to training examples, and the weights are large for the examples that are difficult to classify. Misla- beled examples caused by annotators tend to be difficult examples to classify and these authors conducted error detection of POS tags and PP attachment information in a corpus by extract- ing examples with a large weight. Some probabilistic approaches for …

Content truncated to 500 words