genCNN: A Convolutional Architecture for Word Sequence Prediction

Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing, pages 1567–1576, Beijing, China, July 26-31, 2015. c⃝2015 Association for Computational Linguistics A Convolutional Architecture for Word Sequence Prediction Mingxuan Wang1 Zhengdong Lu2 Hang Li2 Wenbin Jiang1 Qun Liu3,1 1Key Laboratory of Intelligent Information Processing, Institute of Computing Technology, Chinese Academy of Sciences {wangmingxuan,jiangwenbin,liuqun}@ict.ac.cn 2Noah’s Ark Lab, Huawei Technologies {Lu.Zhengdong,HangLi.HL}@huawei.com 3ADAPT Centre, School of Computing, Dublin City University Abstract We propose a convolutional neural net- work, named genCNN, for word se- quence prediction. Different from previous work on neural network- based language modeling and genera- tion (e.g., RNN or LSTM), we choose not to greedily summarize the history of words as a fixed length vector. In- stead, we use a convolutional neural network to predict the next word with the history of words of variable length. Also different from the existing feed- forward networks for language mod- eling, our model can effectively fuse the local correlation and global cor- relation in the word sequence, with a convolution-gating strategy specifi- cally designed for the task. We argue that our model can give adequate rep- resentation of the history, and there- fore can naturally exploit both the short and long range dependencies. Our model is fast, easy to train, and read- ily parallelized. Our extensive exper- iments on text generation and n-best re-ranking in machine translation show that genCNN outperforms the state-of- the-arts with big margins. 1 Introduction Both language modeling (Wu and Khudanpur, 2003; Mikolov et al., 2010; Bengio et al., 2003) and text generation (Axelrod et al., 2011) boil down to modeling the conditional proba- bility of a word given the proceeding words. Previously, it is mostly done through purely memory-based approaches, such as n-grams, which cannot deal with long sequences and has to use some heuristics (called smoothing) for rare ones. Another family of methods are based on distributed representations of words, which is usually tied with a neural-network (NN) ar- chitecture for estimating the conditional prob- abilities of words. Two categories of neural networks have been used for language modeling: 1) recurrent neu- ral networks (RNN), and 2) feedfoward net- work (FFN): • The RNN-based models, including its variants like LSTM, enjoy more popu- larity, mainly due to their flexible struc- tures for processing word sequences of ar- bitrary lengths, and their recent empiri- cal success(Sutskever et al., 2014; Graves, 2013). We however argue that RNNs, with their power built on the recursive use of a relatively simple computation units, are forced to make greedy summarization of the history and consequently not effi- cient on modeling word sequences, which clearly have a bottom-up structures. • The FFN-based models, on the other hand, avoid this difficulty by feeding di- rectly on the history. However, the FFNs are built on fully-connected networks, rendering them inefficient on capturing local structures of languages. Moreover their “rigid” architectures make it futile to handle the great variety of patterns in long range …

Content truncated to 500 words