The University of Edinburgh’s systems submission to the MT task at IWSLT

Integrating Encyclopedic Knowledge into Neural Language Models Yang Zhang, Jan Niehues, Alexander Waibel Institute for Anthropomatics Karlsruhe Institute of Technology, Germany firstname.lastname@kit.edu Abstract Neural models have recently shown big improvements in the performance of phrase-based machine translation. Recur- rent language models, in particular, have been a great suc- cess due to their ability to model arbitrary long context. In this work, we integrate global semantic information extracted from large encyclopedic sources into neural network lan- guage models. We integrate semantic word classes extracted from Wikipedia and sentence level topic information into a recurrent neural network-based language model. The new re- sulting models exhibit great potential in alleviating data spar- sity problems with the additional knowledge provided. This approach of integrating global information is not restricted to language modeling but can also be easily applied to any model that profits from context or further data resources, e.g. neural machine translation. Using this model has improved rescoring quality of a state-of-the-art phrase-based transla- tion system by 0.84 BLEU points. We performed experi- ments on two language pairs. 1. Introduction Recurrent neural network language models (RNNLMs) have recently shown great improvements in statistical machine translation (SMT), both during decoding and rescoring. The use of continuous word representations has achieved bet- ter generalization of the data which effectively lowered data sparsity problems. Furthermore, the recurrent connections are able to model long-range dependencies. Yet, most of these models strictly depend on monolingual and parallel data, which is sometimes not available in large amounts or only for specific domains, especially with low-resource lan- guages. This has motivated RNNLMs to take multiple paral- lel streams of data as input instead of just the single stream of surface words. These so-called factors can be used to add ad- ditional information, e.g. part-of-speech (POS) or automatic word clusters, which helps mainly with morphologically rich languages (e.g. Romanian, German). However, so far the additional factors were only limited to syntactic or local con- text information preceding the current word. Especially for languages without sufficient training data it is important to take advantage of other knowledge sources, e.g. encyclope- dia. For example, Wikipedia has become a growing source for learning general concepts since the emergence of the In- ternet has led to an explosion of textual data. In this paper, we studied the integration of large encyclopedia sources into RNNLMs by using two approaches. In the first approach we use a factored model to integrate Wikipedia categories. In order to understand large unstructured data sets great achievements have been made in latent concept learning in the area of information retrieval. Techniques that categorize documents include latent semantic analysis and probabilistic topic modeling. In the second approach, we use term fre- quency–inverse document frequency (tf-idf), latent semantic analysis (LSA) and latent Dirichlet allocation (LDA) to com- pute a real-valued topic vector for each sentence that is fed as an additional input into the network. These approaches utilize external word categories and topic information in addition to the local contexts implicitly provided by …

Content truncated to 500 words