Sequential Short-Text Classification with Recurrent and Convolutional Neural Networks

Proceedings of NAACL-HLT 2016, pages 515–520, San Diego, California, June 12-17, 2016. c⃝2016 Association for Computational Linguistics Sequential Short-Text Classification with Recurrent and Convolutional Neural Networks Ji Young Lee∗ MIT jjylee@mit.edu Franck Dernoncourt∗ MIT francky@mit.edu Abstract Recent approaches based on artificial neural networks (ANNs) have shown promising re- sults for short-text classification. However, many short texts occur in sequences (e.g., sen- tences in a document or utterances in a dia- log), and most existing ANN-based systems do not leverage the preceding short texts when classifying a subsequent one. In this work, we present a model based on recurrent neural networks and convolutional neural networks that incorporates the preceding short texts. Our model achieves state-of-the-art results on three different datasets for dialog act predic- tion. 1 Introduction Short-text classification is an important task in many areas of natural language processing, includ- ing sentiment analysis, question answering, or dia- log management. Many different approaches have been developed for short-text classification, such as using Support Vector Machines (SVMs) with rule-based features (Silva et al., 2011), combin- ing SVMs with naive Bayes (Wang and Manning, 2012), and building dependency trees with Con- ditional Random Fields (Nakagawa et al., 2010). Several recent studies using ANNs have shown promising results, including convolutional neural networks (CNNs) (Kim, 2014; Blunsom et al., 2014; Kalchbrenner et al., 2014) and recursive neural net- works (Socher et al., 2012). Most ANN systems classify short texts in iso- lation, i.e., without considering preceding short ∗ These authors contributed equally to this work. texts. However, short texts usually appear in se- quence (e.g., sentences in a document or utter- ances in a dialog), and therefore using information from preceding short texts may improve the clas- sification accuracy. Previous works on sequential short-text classification are mostly based on non- ANN approaches, such as Hidden Markov Models (HMMs) (Reithinger and Klesen, 1997; Stolcke et al., 2000; Surendran and Levow, 2006), maximum entropy (Ang et al., 2005), naive Bayes (Lendvai and Geertzen, 2007), and conditional random fields (CRFs) (Kim et al., 2010; Quarteroni et al., 2011). Inspired by the performance of ANN-based sys- tems for non-sequential short-text classification, we introduce a model based on recurrent neural net- works (RNNs) and CNNs for sequential short-text classification, and evaluate it on the dialog act classi- fication task. A dialog act characterizes an utterance in a dialog based on a combination of pragmatic, se- mantic, and syntactic criteria. Its accurate detection is useful for a range of applications, from speech recognition to automatic summarization (Stolcke et al., 2000). Our model achieves state-of-the-art re- sults on three different datasets. 2 Model Our model comprises two parts. The first part gener- ates a vector representation for each short text using either the RNN or CNN architecture, as discussed in Section 2.1 and Figure 1. The second part classifies the current short text based on the vector representa- tions of the current as well as a few preceding short texts, as presented in Section 2.2 and Figure 2. We denote scalars with …

Content truncated to 500 words