Semi-Supervised Sequential Labeling and Segmentation Using Giga-Word Scale Unlabeled Data

Proceedings of ACL-08: HLT, pages 665–673, Columbus, Ohio, USA, June 2008. c⃝2008 Association for Computational Linguistics Semi-Supervised Sequential Labeling and Segmentation using Giga-word Scale Unlabeled Data Jun Suzuki and Hideki Isozaki NTT Communication Science Laboratories, NTT Corp. 2-4 Hikaridai, Seika-cho, Soraku-gun, Kyoto, 619-0237 Japan {jun, isozaki}@cslab.kecl.ntt.co.jp Abstract This paper provides evidence that the use of more unlabeled data in semi-supervised learn- ing can improve the performance of Natu- ral Language Processing (NLP) tasks, such as part-of-speech tagging, syntactic chunking, and named entity recognition. We first pro- pose a simple yet powerful semi-supervised discriminative model appropriate for handling large scale unlabeled data. Then, we describe experiments performed on widely used test collections, namely, PTB III data, CoNLL’00 and ’03 shared task data for the above three NLP tasks, respectively. We incorporate up to 1G-words (one billion tokens) of unlabeled data, which is the largest amount of unlabeled data ever used for these tasks, to investigate the performance improvement. In addition, our results are superior to the best reported re- sults for all of the above test collections. 1 Introduction Today, we can easily find a large amount of un- labeled data for many supervised learning applica- tions in Natural Language Processing (NLP). There- fore, to improve performance, the development of an effective framework for semi-supervised learning (SSL) that uses both labeled and unlabeled data is at- tractive for both the machine learning and NLP com- munities. We expect that such SSL will replace most supervised learning in real world applications. In this paper, we focus on traditional and impor- tant NLP tasks, namely part-of-speech (POS) tag- ging, syntactic chunking, and named entity recog- nition (NER). These are also typical supervised learning applications in NLP, and are referred to as sequential labeling and segmentation problems. In some cases, these tasks have relatively large amounts of labeled training data. In this situation, supervised learning can provide competitive results, and it is difficult to improve them any further by using SSL. In fact, few papers have succeeded in showing significantly better results than state-of-the- art supervised learning. Ando and Zhang (2005) re- ported a substantial performance improvement com- pared with state-of-the-art supervised learning re- sults for syntactic chunking with the CoNLL’00 shared task data (Tjong Kim Sang and Buchholz, 2000) and NER with the CoNLL’03 shared task data (Tjong Kim Sang and Meulder, 2003). One remaining question is the behavior of SSL when using as much labeled and unlabeled data as possible. This paper investigates this question, namely, the use of a large amount of unlabeled data in the presence of (fixed) large labeled data. To achieve this, it is paramount to make the SSL method scalable with regard to the size of unlabeled data. We first propose a scalable model for SSL. Then, we apply our model to widely used test collec- tions, namely Penn Treebank (PTB) III data (Mar- cus et al., 1994) for POS tagging, CoNLL’00 shared task data for syntactic chunking, and CoNLL’03 shared task data for NER. We used …

Content truncated to 500 words