A Parallel Recurrent Neural Network for Language Modeling with POS Tags

31st Pacific Asia Conference on Language, Information and Computation (PACLIC 31), pages 140–147 Cebu City, Philippines, November 16-18, 2017 Copyright c⃝2017 Chao Su, Heyan Huang, Shumin Shi, Yuhang Guo and Hao Wu A Parallel Recurrent Neural Network for Language Modeling with POS Tags Chao Su1,3, Heyan Huang1,2, Shumin Shi1,2,∗, Yuhang Guo1, Hao Wu1 1School of Computer Science and Technology, Beijing Institute of Technology, Beijing 100081, China 2Beijing Engineering Research Center of High Volume Language Information Processing and Cloud Computing Applications, Beijing 100081, China 3Beijing Advanced Innovation Center for Imaging Technology, Capital Normal University, Beijing 100048, China {suchao,hhy63,bjssm,guoyuhang,wuhao123}@bit.edu.cn Abstract Language models have been used in many nat- ural language processing applications. In re- cent years, the recurrent neural network based language models have defeated the conven- tional n-gram based techniques. However, it is difficult for neural network architectures to use linguistic annotations. We try to incorpo- rate part-of-speech features in recurrent neural network language model, and use them to pre- dict the next word. Specifically, we proposed a parallel structure which contains two recurrent neural networks, one for word sequence mod- eling and another for part-of-speech sequence modeling. The state of part-of-speech net- work helped improve the word sequence’s pre- diction. Experiments show that the proposed method performs better than the traditional re- current network on perplexity and is better at reranking machine translation outputs.1 1 Introduction Language models (LMs) are crucial parts of many natural language processing applications, such as automatic speech recognition, statistical machine translation, and natural language generation. Lan- guage modeling aims to predict the next word given context or to give the probability of a word sequence in textual data. In the past decades, n-gram based modeling techniques were most commonly used in such NLP applications. However, the recurrent neu- ral network based language model (RNNLM) and ∗Corresponding author 1Our code is available at https://github.com/ chao-su/prnnlm its extensions (Mikolov et al., 2010; Mikolov et al., 2011) have received a lot of attention and achieved the new state of the art results since 2010. The most important advantage of RNNLM is that it has the potential to model unlimited size of context, due to its recurrent property. That is to say, the hidden layer has a recurrent connection to itself at previous timestep. Part-of-speech (POS) tags capture the syntactic role of each word, and has been proved to be use- ful for language modeling (Kneser and Ney, 1993; A. Heeman, 1998; Galescu and Ringger, 1999; Wang and Harper, 2002). Jelinek (1985) pointed out that we can replace the classes with POS tags in lan- guage model. Kneser and Ney (1993) incorporated POS tags into n-gram LM and got 37 percents im- provement. But they got only 10 percents improve- ment with classes through clustering. A. Heeman (1998) redefined the objective of automatic speech recognition: to get both the word sequence and the POS sequence. His experiments showed 4.2 percent reduction on perplexity over classes. It is common to build probabilistic graphical models using many different linguistic annotations (Finkel …

Content truncated to 500 words