Discriminative Lexicon Adaptation for Improved Character Accuracy - A New Direction in Chinese Language Modeling

Proceedings of the 47th Annual Meeting of the ACL and the 4th IJCNLP of the AFNLP, pages 755–763, Suntec, Singapore, 2-7 August 2009. c⃝2009 ACL and AFNLP Discriminative Lexicon Adaptation for Improved Character Accuracy – A New Direction in Chinese Language Modeling Yi-cheng Pan Speech Processing Labratory National Taiwan University Taipei, Taiwan 10617 thomashughPan@gmail.com Lin-shan Lee Speech Processing Labratory National Taiwan University Taipei, Taiwan 10617 lsl@speech.ee.ntu.edu.tw Sadaoki Furui Furui Labratory Tokyo Institute of Technology Tokyo 152-8552 Japan furui@furui.cs.titech.ac.jp Abstract While OOV is always a problem for most languages in ASR, in the Chinese case the problem can be avoided by utilizing char- acter n-grams and moderate performances can be obtained. However, character n- gram has its own limitation and proper addition of new words can increase the ASR performance. Here we propose a dis- criminative lexicon adaptation approach for improved character accuracy, which not only adds new words but also deletes some words from the current lexicon. Different from other lexicon adaptation approaches, we consider the acoustic features and make our lexicon adaptation criterion consistent with that in the decoding process. The pro- posed approach not only improves the ASR character accuracy but also significantly enhances the performance of a character- based spoken document retrieval system. 1 Introduction Generally, an automatic speech recognition (ASR) system requires a lexicon. The lexicon defines the possible set of output words and also the building units in the language model (LM). Lexical words offer local constraints to combine phonemes into short chunks while the language model combines phonemes into longer chunks by more global con- straints. However, it’s almost impossible to include all words into a lexicon both due to the technical difficulty and also the fact that new words are cre- ated continuously. The missed out words will never be recognized, which is the well-known OOV prob- lem. Using graphemes for OOV handling is pro- posed in English (Bisani and Ney, 2005). Although this sacrifices some of the lexical constraints and in- troduces a further difficulty to combine graphemes back into words, it is compensated by its ability for 5.8K characters 61.5K full lexicon bigram 63.55% 73.8% trigram 74.27% 79.28% Table 1: Character recognition accuracy under dif- ferent lexicons and the order of language model. open vocabulary ASR. Morphs are another possi- bility, which are longer than graphemes but shorter than words, in other western languages (Hirsim¨aki et al., 2005). Chinese language, on the other hand, is quite different from western languages. There are no blanks between words and the definition for words is vague. Since almost all characters in Chinese have their own meanings and words are composed of the characters, there is an obvious solution for the OOV problem: simply using all characters as the lexicon. In Table 1 we see the differences in character recognition accuracy by using only 5.8K characters and a full set of 61.5K lexicon. The train- ing set and testing set are the same as those that will be introduced in Section 4.1. It is clear …

Content truncated to 500 words