Inducing Word and Part-of-Speech with Pitman-Yor Hidden Semi-Markov Models

Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing, pages 1774–1782, Beijing, China, July 26-31, 2015. c⃝2015 Association for Computational Linguistics Inducing Word and Part-of-Speech with Pitman-Yor Hidden Semi-Markov Models Kei Uchiumi Hiroshi Tsukahara Denso IT Laboratory, Inc. Shibuya Cross Tower 28F 2-15-1 Shibuya, Tokyo, Japan {kuchiumi,htsukahara}@d-itlab.co.jp Daichi Mochihashi The Institute of Statistical Mathematics 10-3 Midori-cho, Tachikawa city Tokyo, Japan daichi@ism.ac.jp Abstract We propose a nonparametric Bayesian model for joint unsupervised word seg- mentation and part-of-speech tagging from raw strings. Extending a previous model for word segmentation, our model is called a Pitman-Yor Hidden Semi- Markov Model (PYHSMM) and consid- ered as a method to build a class n-gram language model directly from strings, while integrating character and word level information. Experimental results on stan- dard datasets on Japanese, Chinese and Thai revealed it outperforms previous re- sults to yield the state-of-the-art accura- cies. This model will also serve to analyze a structure of a language whose words are not identified a priori. 1 Introduction Morphological analysis is a staple of natural lan- guage processing for broad languages. Especially for some East Asian languages such as Japanese, Chinese or Thai, word boundaries are not explic- itly written, thus morphological analysis is a cru- cial first step for further processing. Note that also in Latin and old English, scripts were orig- inally written with no word indications (scripta continua), but people felt no difficulty reading them. Here, morphological analysis means word segmentation and part-of-speech (POS) tagging. For this purpose, supervised methods have of- ten been employed for training. However, to train such supervised classifiers, we have to pre- pare a large amount of training data with cor- rect annotations, in this case, word segmentation and POS tags. Creating and maintaining these data is not only costly but also very difficult, be- cause generally there are no clear criteria for ei- ther “correct” segmentation or POS tags. In fact, since there are different standards for Chinese word segmentation, widely used SIGHAN Bake- off dataset (Emerson, 2005) consists of multiple parts employing different annotation schemes. Lately, this situation has become increasingly important because there are strong demands for processing huge amounts of text in consumer gen- erated media such as Twitter, Weibo or Facebook (Figure 1). They contain a plethora of colloquial expressions and newly coined words, including sentiment expressions such as emoticons that can- not be covered by fixed supervised data. To automatically recognize such linguistic phe- nomena beyond small “correct” supervised data, we have to extract linguistic knowledge from the statistics of strings themselves in an unsupervised fashion. Needless to say, such methods will also contribute to analyzing speech transcripts, classic texts, or even unknown languages. From a scien- tific point of view, it is worth while to find “words” and their part-of-speech purely from a collection of strings without any preconceived assumptions. To achieve that goal, there have been two kinds of approaches: heuristic methods and statisti- …

Content truncated to 500 words