Automatic Extraction of New Words from Japanese Texts using Generalized Forward-Backward Search

Automatic Extraction of New Words from Japanese Texts using Generalized Forward-Backward Search Masaaki NAGATA NTT Information and Communication Systems Laboratories 1-2356 Take, Yokosuka-Shi, Kanagawa, 238-03 Japan nagat a©nttnly. ± sl. ntt. j p Abstract We present a novel new word extraction method from Japanese texts based on expected word frequencies. First, we compute expected word frequencies from Japanese texts using a robust stochastic N-best word segmenter. We then ex- tract new words by filtering out erroneous word hypotheses whose expected word frequencies are lower than the predefined threshold. The method is derived from an approximation of the general- ized version of the Forward-Backward algorithm. When the Japanese word segmenter is trained on a 4.7 million word segmented corpus and tested on 1000 sentences whose out-of-vocabulary rate is 2.1%, the accuracy of the new word extraction method is 43.7% recall and 52.3% precision. Introduction Segmentation of sentences into words is trivial in English because words are delimited by spaces. It is a simple task to count word frequencies in a given text. It is also a simple task to list all new words (unknown words), namely, the words in a given text that are not found in the system dictionary. However, several languages such as Japanese, Chinese and Thai do not put spaces between words and so in these languages word segmentation, word frequency counting, and new word extraction remain unsolved problems in com- putational linguistics. Most Japanese NLP applications require word segmentation as a first stage because there are phonological units and semantic units whose pro- nunciation and/or meaning is not trivially deriv- able from the pronunciation and/or meaning of the individual characters. It is well known that the accuracy of word segmentation greatly depends on the coverage of the dictionary, in other words, the Out-Of-Vocabulary (00V) rate of the target texts. Our goal is to provide a method to automati- cally extract new words from Japanese texts. This nmthod should adapt the dictionary of the word segmenter to new domains and applications. It should also maintain the dictionary by collecting new words in the target domain. The applica- tion of the word segmenter is described elsewhere (Nagata, 1996). The approach we take is as follows: First, we design a statistical language model that can as- sign a reasonable word probability to an arbitrary substring in the input sentence, whether or not it is truly a word. Second, we devised a method to obtain the expected word N-gram count in the target texts, using an N-best word segmentation algorithm (Nagata, 1994). Finally, we extract new words by filtering out spurious word hypotheses whose expected word frequencies are lower than the threshold. Japanese Morphological Analysis Before we start, we briefly explain the difficul- ties of Japanese morphological analysis, especially when the input sentence includes unknown words. Suppose the input sentence is "-~4)~p/~7 ~}~ ENIAC 69 50 ~ 3o ", which means "University of Pennsylvania celebrates the 50th anniversary of ENIAC", where the words ~Y5~ JP/~7 (transliteration of 'Pennsylvania') and ENIAC (the …

Content truncated to 500 words