Fast Gated Neural Domain Adaptation: Language Model as a Case Study

Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 1386–1397, Osaka, Japan, December 11-17 2016. Fast Gated Neural Domain Adaptation: Language Model as a Case Study Jian Zhang, Xiaofeng Wu, Andy Way, Qun Liu ADAPT Centre School of Computing Dublin City University, Ireland jian.zhang,xiaofeng.wu,andy.way,qun.liu@adaptcentre.ie Abstract Neural network training has been shown to be advantageous in many natural language processing applications, such as language modelling or machine translation. In this paper, we describe in detail a novel domain adaptation mechanism in neural network training. Instead of learning and adapting the neural network on millions of training sentences – which can be very time- consuming or even infeasible in some cases – we design a domain adaptation gating mechanism which can be used in recurrent neural networks and quickly learn the out-of-domain knowledge directly from the word vector representations with little speed overhead. In our experiments, we use the recurrent neural network language model (LM) as a case study. We show that the neural LM perplexity can be reduced by 7.395 and 12.011 using the proposed domain adaptation mechanism on the Penn Treebank and News data, respectively. Furthermore, we show that using the domain-adapted neural LM to re-rank the statistical machine translation n-best list on the French-to-English language pair can significantly improve translation quality. 1 Introduction One challenge which rises above others in natural language processing (NLP) is where application per- formance decreases when there are dissimilarities between the training and the testing environments. In research on domain adaptation, training data with the same style and topic (van der Wees et al., 2015) as the test data is often defined as in-domain (ID) data, with everything else called general-domain (GD) data. The ID data is often scarce and expensive to obtain, whereas the GD is plentiful and easy to access. One approach to address the situation of scarce ID training data is through data selection. For example, using the perplexity difference between ID and GD can select data that is close to ID and away from GD (Moore and Lewis, 2010). The selected data can then be concatenated with ID data for training. However, the drawback of using data selection is a threshold needs to be set, which often requires many models to be trained and evaluated. The situation will worsen if neural networks are used since the computational cost is immense in neural network training, where models often require days to converge even with the help of GPU-accelerated computing. Thus, simply adapting previous domain adaptation techniques into the neural network framework may not be efficient or effective. Ideally, we want to have an adaptation approach which has the ability to learn knowledge from huge corpora at speed. The question that arises here is how to make use of large amounts of GD data but avoiding long training times. In neural network training, words are represented as distributed representations, so-called “word vec- tors”, which can be pre-trained or trained with a specific task in mind. Although a …

Content truncated to 500 words