Generation of Adaptive Vocabulary Lexicon for Japanese LVCSR

GENERATION OF ADAPTIVE VOCABULARY LEXICON FOR JAPANESE LVCSR Charles C.H. Jie Philips Innovation Center, Taipei 24F, 66, Sec. 1 Chung-Hsiao W. Rd Taipei, Taiwan, R.O.0 charles.jie @philips.com or chjie @ms23.url.com.tw ABSTRACT One of the thorniest problems of large vocabulary continuous speech recognition systems is the large number of out-of-vocabulary (OOV) words. This is especially the case for the languages like Japanese, which has many inflections, compound words and loanwords. The OOV words vary with the application domains. It's not realistic to have a big general-purpose lexicon including any possible 00V words. Furthermore, embedded speech recognition systems become more and more popular recently. They strongly demand an economical and effective exploitation of lexicon space. In this paper, we introduce a lexicon development model dealing with different kinds of OOV words with the help of linguistic morphological knowledge. It provides unsupervised, fast vocabulary adaptation among different application domains. In the experiments that adapt a lexicon of typical vocabulary, we were able to reduce the OOV rate by 40% and improve the word segmentation error rate by 27%. And the smaller the lexicon is, the more we benefit from the vocabulary adaptation. 1. INTRODUCTION No sooner had we created our first lexicon for Japanese large vocabulary continuous speech recognition than we realized the challenge of out-of-vocabulary problems involved. Although 220 thousand words have been acquired from traditional dictionaries to build our lexicon, we can still find ten thousands of 00V words and one million occurrences of them in our corpora. The high OOV rate, ranging from 4.3% to 9.8%, may cause an impact on recognition system and account for a major share of recognition errors. Inspecting to the text segmentation errors, several types of vocabulary problems can be identified. As an agglutinative language, the inflection of Japanese is very complicated and hard to handle. For example the gerund form `natte' is one of the difficult problems for segmentation'. About compound and derived words, it's difficult to make new words recognizable without introducing new composing units (sub- words). We need to acquire prefixes and a suffixes from the derived words like Ifi-Atl-'(new food law) to help identify new words composed in the similar way. Again we sometimes have to combine words to make a new entry in lexicon because the pronunciation of the combination can not be predicted by individual components, such as ' — '(one minute). We also found that Arabic numbers, alphabetic acronyms, and katakana2 words are other prevailing types of OOV words appearing in news context like the Nikkei News3. However, not all of the OOV words should be added into the lexicon. We have to consider certain selection strategy to keep the lexicon efficient. An LVCSR system will have difficulty to perform well if it fails to cope with the vocabulary problems in early stages. We consider an integrated approach to deal with the above issues. Motivated by Geutner's work[1] on similar problems and based on our needs, we propose a lexicon development model to easily and quickly To keep …

Content truncated to 500 words