Very Deep Convolutional Networks for Text Classification

Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 1107–1116, Valencia, Spain, April 3-7, 2017. c⃝2017 Association for Computational Linguistics Very Deep Convolutional Networks for Text Classification Alexis Conneau Facebook AI Research aconneau@fb.com Holger Schwenk Facebook AI Research schwenk@fb.com Yann Le Cun Facebook AI Research yann@fb.com Lo¨ıc Barrault LIUM, University of Le Mans, France loic.barrault@univ-lemans.fr Abstract The dominant approach for many NLP tasks are recurrent neural networks, in par- ticular LSTMs, and convolutional neural networks. However, these architectures are rather shallow in comparison to the deep convolutional networks which have pushed the state-of-the-art in computer vi- sion. We present a new architecture (VD- CNN) for text processing which operates directly at the character level and uses only small convolutions and pooling oper- ations. We are able to show that the per- formance of this model increases with the depth: using up to 29 convolutional layers, we report improvements over the state-of- the-art on several public text classification tasks. To the best of our knowledge, this is the first time that very deep convolutional nets have been applied to text processing. 1 Introduction The goal of natural language processing (NLP) is to process text with computers in order to analyze it, to extract information and eventually to rep- resent the same information differently. We may want to associate categories to parts of the text (e.g. POS tagging or sentiment analysis), struc- ture text differently (e.g. parsing), or convert it to some other form which preserves all or part of the content (e.g. machine translation, summariza- tion). The level of granularity of this processing can range from individual characters to subword units (Sennrich et al., 2016) or words up to whole sentences or even paragraphs. After a couple of pioneer works (Bengio et al. (2001), Collobert and Weston (2008), Collobert et al. (2011) among others), the use of neural net- works for NLP applications is attracting huge in- terest in the research community and they are sys- tematically applied to all NLP tasks. However, while the use of (deep) neural networks in NLP has shown very good results for many tasks, it seems that they have not yet reached the level to outperform the state-of-the-art by a large margin, as it was observed in computer vision and speech recognition. Convolutional neural networks, in short Con- vNets, are very successful in computer vision. In early approaches to computer vision, handcrafted features were used, for instance “scale-invariant feature transform (SIFT)”(Lowe, 2004), followed by some classifier. The fundamental idea of Con- vNets(LeCun et al., 1998) is to consider feature extraction and classification as one jointly trained task. This idea has been improved over the years, in particular by using many layers of convolutions and pooling to sequentially extract a hierarchical representation(Zeiler and Fergus, 2014) of the in- put. The best networks are using more than 150 layers as in (He et al., 2016a; He et al., 2016b). Many NLP approaches consider words as ba- …

Content truncated to 500 words