Unsupervised phonemic Chinese word segmentation using Adaptor Grammars

Proceedings of the 23rd International Conference on Computational Linguistics (Coling 2010), pages 528–536, Beijing, August 2010 Unsupervised phonemic Chinese word segmentation using Adaptor Grammars Mark Johnson Department of Computing Macquarie University Mark.Johnson@mq.edu.au Katherine Demuth Department of Linguistics Macquarie University Katherine.Demuth@mq.edu.au Abstract Adaptor grammars are a framework for expressing and performing inference over a variety of non-parametric linguistic models. These models currently provide state-of-the-art performance on unsuper- vised word segmentation from phonemic representations of child-directed unseg- mented English utterances. This paper in- vestigates the applicability of these mod- els to unsupervised word segmentation of Mandarin. We investigate a wide vari- ety of different segmentation models, and show that the best segmentation accuracy is obtained from models that capture inter- word “collocational” dependencies. Sur- prisingly, enhancing the models to exploit syllable structure regularities and to cap- ture tone information does improve over- all word segmentation accuracy, perhaps because the information these elements convey is redundant when compared to the inter-word dependencies. 1 Introduction and previous work The word-segmentation task is an abstraction of part of the problem facing a child learning its na- tive language. Fluent speech, even the speech di- rected at children, doesn’t come with silence or pauses delineating acoustic words the way that spaces separate orthographic words in writing sys- tems like that of English. Instead, as most people listening to a language they don’t understand can attest, words in fluent speech “run together”, and a language user needs to learn how to segment utter- ances of the language they are learning into words. This kind of word segmentation is presumably an important first step in acquiring a language. It is scientifically interesting to know what informa- tion might be useful for word segmentation, and just how this information might be used. These scientific questions have motivated a body of re- search on computational models of word segmen- tation. Since as far as we can tell any child can learn any human language, our goal is to develop a single model that can learn to perform accurate word segmentation given input from any human language, rather than a model that specialised to perform well on a single language. This paper extends the previous work on word segmentation by investigating whether one class of models that work very well with English input also work with Chinese input. These models will permit us to study the role that syllable structure constraints and tone in Chinese might play in word segmenta- tion. While learners and fluent speakers undoubt- edly use a wide variety of cues to perform word segmentation, computational models since El- man (1990) have tended to focus on the use of phonotactic constraints (e.g., syllable-structure constrains) and distributional information. Brent and Cartwright (1996) introduced the standard form of the word segmentation task still studied to- day. They extracted the orthographic representa- tions of child-directed speech from the Bernstein- Ratner corpus (Bernstein-Ratner, 1987) and “pho- nologised” them by looking up each word in a pronouncing dictionary. For example, the or- thographic …

Content truncated to 500 words