Modeling Chinese Documents with Topical Word-Character Models

Proceedings of the 22nd International Conference on Computational Linguistics (Coling 2008), pages 345–352 Manchester, August 2008 Modeling Chinese Documents with Topical Word-Character Models Wei Hu1 Nobuyuki Shimizu2 1Department of Computer Science Shanghai Jiao Tong University Shanghai, China 200240 {no_bit,hysheng} @sjtu.edu.cn Hiroshi Nakagawa2 Huanye Sheng1 2Information Technology Center The University of Tokyo Tokyo, Japan 113-0033 {shimizu, nakagawa} @r.dl.itc.u-tokyo.ac.jp Abstract As Chinese text is written without word boundaries, effectively recognizing Chi- nese words is like recognizing colloca- tions in English, substituting characters for words and words for collocations. However, existing topical models that in- volve collocations have a common limi- tation. Instead of directly assigning a top- ic to a collocation, they take the topic of a word within the collocation as the topic of the whole collocation. This is unsatis- factory for topical modeling of Chinese documents. Thus, we propose a topical word-character model (TWC), which al- lows two distinct types of topics: word topic and character topic. We evaluated TWC both qualitatively and quantitatively to show that it is a power- ful and a promising topic model. 1 Introduction Topic models (Blei et al., 2003; Griffiths & Steyvers 2004, 2007) are a class of statistical models in which documents are expressed as mixtures of topics, where a topic is a probability distribution over words. A topic model is a gen- erative model for documents: it specifies a prob- abilistic procedure for generating documents. To make a new document, we choose a distribution over topics. Then, for each word in this docu- ment, we randomly select a topic from the distri- bution, and draw a word from the topic. Once we have a topic model, we can invert the generating process, inferring the set of topics that was re- sponsible for generating a collection of docu- © 2008. Licensed under the Creative Commons Attri- bution-Noncommercial-Share Alike 3.0 Unported license (http://creativecommons.org/licenses/by-nc- sa/3.0/). Some rights reserved. ments. Although most topic models treat a document as a bag-of-words, the assumption has obvious shortcomings. Suppose there are many docu- ments about art and musicals in New York. Then, we find a topic represented by words such as “art”, “musical”, and “New”, instead of getting “New York”. The bag-of-words assumption makes the topic model split a collocation—a phrase with meaning beyond the individual words—into individual words that have a differ- ent meaning. One example of a collocation is the phrase “white house”. In politics, it carries a spe- cial meaning beyond a house that is white, whereas “yellow house” does not. While it is reasonable for English, the equiva- lent bag-of-characters assumption is especially troublesome for modeling Chinese documents, where almost all basic vocabularies are the equi- valents of English collocations. In Chinese, some of the most commonly used couple-of-thousand characters are combined to make up a word, and no word boundary is given in the text. Effectively, Chinese words are like collocations in English. The difficulty is that there are over- whelmingly more of them, enough to render a bag-of-character assumption unreasonable for Chinese. Therefore, …

Content truncated to 500 words