Using Factored Word Representation in Neural Network Language Models

Proceedings of the First Conference on Machine Translation, Volume 1: Research Papers, pages 74–82, Berlin, Germany, August 11-12, 2016. c⃝2016 Association for Computational Linguistics Using Factored Word Representation in Neural Network Language Models Jan Niehues, Thanh-Le Ha, Eunah Cho and Alex Waibel Institute for Anthropomatics Karlsruhe Institute of Technology, Germany firstname.secondname@kit.edu Abstract Neural network language and translation models have recently shown their great po- tentials in improving the performance of phrase-based machine translation. At the same time, word representations using dif- ferent word factors have been translation quality and are part of many state-of-the- art machine translation systems. used in many state-of-the-art machine translation systems, in order to support better transla- tion quality. In this work, we combined these two ideas by investigating the combination of both techniques. By representing words in neu- ral network language models using differ- ent factors, we were able to improve the models themselves as well as their impact on the overall machine translation perfor- mance. This is especially helpful for mor- phologically rich languages due to their large vocabulary size. Furthermore, it is easy to add additional knowledge, such as source side information, to the model. Using this model we improved the trans- lation quality of a state-of-the-art phrase- based machine translation system by 0.7 BLEU points. We performed experiments on three language pairs for the news trans- lation task of the WMT 2016 evaluation. 1 Introduction Recently, neural network models are deployed ex- tensively for better translation quality of statisti- cal machine translation (Le et al., 2011; Devlin et al., 2014). For the language model as well as for the translation model, neural network-based mod- els showed improvements when used during de- coding as well as when used in re-scoring. In phrase-based machine translation (PBMT), word representation using different factors (Koehn and Hoang, 2007) are commonly used in state- of-the-art systems. Using Part-of-Speech (POS) information or automatic word clusters is es- pecially important for morphologically rich lan- guages which often have a large vocabulary size. Language models based on these factors are able to consider longer context and therefore improve the modelling of the overall structure. Further- more, the POS information can be used to improve the modelling of word agreement, which is often a difficult task when handling morphologically rich languages. Until now, word factors have been used rela- tively limited in neural network models. Auto- matic word classes have been used to structure the output layer (Le et al., 2011) and as input in feed forward neural network language models (Niehues and Waibel, 2012). In this work, we propose a multi-factor recur- rent neural network (RNN)-based language model that is able to facilitate all available information about the word in the input as well as in the out- put. We evaluated the technique using the surface form, POS-tag and automatic word clusters using different cluster sizes. Using this model, it is also possible to integrate source side information into the model. By using the model as a bilingual model, the probability …

Content truncated to 500 words