What makes a word: Learning base units in Japanese for speech recognition

What makes a word: Learning base units in Japanese for speech recognition Laura Mayfield Tomokiyo Language Technology Institute Carnegie Mellon University 4910 Forbes Avenue Pittsburgh, PA 15213, USA laura@as, cmu. edu Klaus Ries Universit£t Karlsruhe Fakult/it ffir Informatik Interactive Systems Laboratories 76128 Karlsruhe, Germany kries@ira, uka. de Abstract We describe an automatic process for learn- ing word units in Japanese. Since the Japanese orthography has no spaces de- limiting words, the first step in building a Japanese speech recognition system is to define the units that will be recognized. Our method applies a compound-finding algorithm, previously used to find word se- quences in English, to learning syllable se- quences in Japanese. We report that we were able not only to extract meaningful units, eliminating the need for possibly in- consistent manual segmentation, but also to decrease perplexity using this automatic procedure, which relies on a statistical, not syntactic, measure of relevance. Our al- gorithm also uncovers the kinds of envi- ronments that help the recognizer predict phonological alternations, which are often hidden by morphologically-motivated tok- enization. 1 Introduction What defines a word when there are no spaces in a written language? Words, as they are known in English and other western languages, are the basic units of recognition in most CSR systems, but when a language is written as a string of characters with no white space, how does one go about specifying the fundamental units that must be recognized? Map- ping onto English-style words is one solution, but an artificial one, and may hide natural character- istics of Japanese that can be important in recog- nition. Recognizing phonemes, or short phoneme clusters, is another option, but recognition accuracy can improve when we have longer phoneme strings to work with; acoustic confusability decreases and a long word is a more useful predictor of subsequent words than a single syllable. Automatic segmenting tools eliminate an often inconsistent manual segmen- tation step, but are generally based on morpholog- ical analysis, which can produce units smaller than are desirable for speech recognition. Certainly, there exist words as can be looked up in a dictionary, but when a language is as heavily inflected as Japanese is, that only solves part of the problem. In this pa- per we describe an automatic process for learning base units in Japanese and discuss its usefulness for speech recognition. 2 The problem with Japanese The Japanese language is written without spaces in between words. This means that before one can even start designing a recognition or translation sys- tem for Japanese the units that will be recognized, or translated, must be defined. Many sequences of phonemes, particularly those representing nouns, are clearly independent and can be designated as free- standing units. Japanese has a rich and fusional inflectional system, though, and delimiting where a verb ending ends and another begins, for example, is seldom straightforward. Japanese has typically been segmented in vari- ations on four ways for the purposes of recogni- tion and parsing, although since many papers …

Content truncated to 500 words