Convolutional Neural Network Language Models

Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1153–1162, Austin, Texas, November 1-5, 2016. c⃝2016 Association for Computational Linguistics Convolutional Neural Network Language Models Ngoc-Quan Pham and German Kruszewski and Gemma Boleda Center for Mind/Brain Sciences University of Trento {firstname.lastname}@unitn.it Abstract Convolutional Neural Networks (CNNs) have shown to yield very strong results in several Computer Vision tasks. Their application to language has received much less attention, and it has mainly focused on static classifica- tion tasks, such as sentence classification for Sentiment Analysis or relation extraction. In this work, we study the application of CNNs to language modeling, a dynamic, sequential prediction task that needs models to capture local as well as long-range dependency infor- mation. Our contribution is twofold. First, we show that CNNs achieve 11-26% better absolute performance than feed-forward neu- ral language models, demonstrating their po- tential for language representation even in se- quential tasks. As for recurrent models, our model outperforms RNNs but is below state of the art LSTM models. Second, we gain some understanding of the behavior of the model, showing that CNNs in language act as feature detectors at a high level of abstraction, like in Computer Vision, and that the model can prof- itably use information from as far as 16 words before the target. 1 Introduction Convolutional Neural Networks (CNNs) are the family of neural network models that feature a type of layer known as the convolutional layer. This layer can extract features by convolving a learnable filter (or kernel) along different positions of a vectorial in- put. CNNs have been successfully applied in Com- puter Vision in many different tasks, including ob- ject recognition, scene parsing, and action recogni- tion (Gu et al., 2015), but they have received less attention in NLP. They have been somewhat ex- plored in static classification tasks where the model is provided with a full linguistic unit as input (e.g. a sentence) and classes are treated as independent of each other. Examples of this are sentence or docu- ment classification for tasks such as Sentiment Anal- ysis or Topic Categorization (Kalchbrenner et al., 2014; Kim, 2014), sentence matching (Hu et al., 2014), and relation extraction (Nguyen and Grish- man, 2015). However, their application to sequen- tial prediction tasks, where the input is construed to be part of a sequence (for example, language model- ing or POS tagging), has been rather limited (with exceptions, such as Collobert et al. (2011)). The main contribution of this paper is a systematic evalu- ation of CNNs in the context of a prominent sequen- tial prediction task, namely, language modeling. Statistical language models are a crucial compo- nent in many NLP applications, such as Automatic Speech Recognition, Machine Translation, and In- formation Retrieval. Here, we study the problem under the standard formulation of learning to predict the upcoming token given its previous context. One successful approach to this problem relies on count- ing the number of occurrences of n-grams while using smoothing and back-off techniques …

Content truncated to 500 words