Factored Language Model based on Recurrent Neural Network

Proceedings of COLING 2012: Technical Papers, pages 2835–2850, COLING 2012, Mumbai, December 2012. Factored Language Model based on Recurrent Neural Network Youzheng Wu Xugang Lu Hitoshi Yamamoto Shigeki Matsuda Chiori Hori Hideki Kashioka National Institute of Information and Communications Technology (NiCT) 3-5 Hikari-dai, Seika-cho, Soraku-gun, Kyoto, Japan, 619-0289 {youzheng.wu,xugang.lu,hitoshi.yamamoto,shigeki.matsuda}@nict.go.jp {chiori.hori,hideki.kashioka}@nict.go.jp Abstract Among various neural network language models (NNLMs), recurrent neural network-based lan- guage models (RNNLMs) are very competitive in many cases. Most current RNNLMs only use one single feature stream, i.e., surface words. However, previous studies proved that language models with additional linguistic information achieve better performance. In this study, we extend RNNLM by explicitly integrating additional linguistic information, including morphological, syn- tactic, or semantic factors. Our proposed RNNLM is called a factored RNNLM that is expected to enhance RNNLMs. A number of experiments are carried out that show the factored RNNLM im- proves the performance for all considered tasks: consistent perplexity and word error rate (WER) reductions. In the Penn Treebank corpus, the relative improvements over n-gram LM and RNNLM are 29.0% and 13.0%, respectively. In the IWSLT-2011 TED ASR test set, absolute WER reduc- tions over RNNLM and n-gram LM reach 0.63 and 0.73 points. Title and Abstract in another language (Chinese) � � �� � �� � �� � �� � �� � �� � �� � �� � �� � �� � �� � �� � �� � �� � � ������������NNLM��������������RNNLM��� ��������������������RNNLM��������������� ��������������������������������������� ������������������RNNLM����������������� ���������fRNNLM����������������fRNNLM������ �������RNNLM���������������������WER����� ����fRNNLM���������n������RNNLM�����29.0%�13.0%� �IWSLT-2011 TED���������fRNNLM�����������0.63��� �n�������0.73����RNNLM�� Keywords: Factored Language Model, Recurrent Neural Network, Large Vocabulary Continuous Speech Recognition. Keywords in Chinese: ������, ������, ��������. 2835 1 Introduction Language models (LM) are a critical component of many application systems such as automatic speech recognition (ASR), machine translation (MT) and optical character recognition (OCR). In the past, statistical back-off n-gram language models with sophisticated smoothing techniques have gained great popularity because of their simplicity and good performance. In n-gram language models, words are represented in a discrete space: the vocabulary. Standard back-off n-gram lan- guage models predict the following word based on the previous n-1 words, which can be expressed as, p(wi|wi−1,..., w1) ≈ p(wi|wi−n+1,..., wi−2, wi−1) (1) Even though n is usually limited to three or four, the number of parameters in a back-off n-gram LM is still enormous. Assuming the vocabulary size is 64K, a 4-gram language model needs to estimate 64K2 bigrams, 64K3 trigrams and 64K4 4-grams. Due to data sparseness, many are not observed during the training phase. This means that n-gram LMs have poor generalization to low-frequency and unseen n-grams. This problem becomes more severe as the vocabulary size increases. Many interesting approaches have been proposed to overcome it in large vocabulary continuous speech recognition (LVCSR) and statistical machine translation systems, especially smoothing techniques (Chen and Goodman, 1996), class n-gram language models (Brown et al., 1992), topic language models (Gildea and Hofmann, 1999; Hsu and Glass, 2006), structured lan- guage models (Chelba and Jelinek, 2000), maximum entropy language models (Rosenfeld, 1996) and random forests language models (Xu and …

Content truncated to 500 words