Tohoku at SemEval-2016 Task 6: Feature-based Model versus Convolutional Neural Network for Stance Detection

Proceedings of SemEval-2016, pages 401–407, San Diego, California, June 16-17, 2016. c⃝2016 Association for Computational Linguistics Tohoku at SemEval-2016 Task 6: Feature-based Model versus Convolutional Neural Network for Stance Detection Yuki Igarashi Tohoku University yuki.i@dc.tohoku.ac.jp Hiroya Komatsu Tohoku University h-komatsu@ecei.tohoku.ac.jp Sosuke Kobayashi Tohoku University sosuke.k@ecei.tohoku.ac.jp Naoaki Okazaki Tohoku University okazaki@ecei.tohoku.ac.jp Kentaro Inui Tohoku University inui@ecei.tohoku.ac.jp Abstract In this paper, we compare feature-based and Neural Network-based approaches on the su- pervised stance classification task for tweets in SemEval-2016 Task 6 Subtask A (Moham- mad et al., 2016). In the feature-based ap- proach, we use external resources such as lex- icons and crawled texts. The Neural Network based approach employs Convolutional Neu- ral Network (CNN). Our results show that the feature-based model outperformed the CNN model on the test data although the CNN model was better than the feature-based model in the cross validation on the training data. 1 Introduction To solve supervised short text classification tasks, there are two major approaches; feature-based and Neural Network based approaches. In traditional feature-based approaches, we extract various fea- tures from a text. The features are usually con- structed from n-grams (e.g., bigrams) of the texts and external resources such as lexicons and unla- beled corpora. In Neural Network based approaches, a number of models for text classifications exist; for exam- ple, Feed-Forward Neural Network model using an average of embeddings of target word sequences as the input layer (Iyyer et al., 2015), Recursive Neural Network (Socher et al., 2011; Socher et al., 2013), and Convolutional Neural Network (CNN) (Johnson and Zhang, 2015; dos Santos and Gatti, 2014; Kim, 2014). In this paper, we compare feature-based and Neural Network based approaches on the super- vised stance classification task for tweets, SemEval- 2016 Task 6 Subtask A (Mohammad et al., 2016). The feature-based approach classifies tweets using logistic regression model. The features are ex- tracted using external knowledge such as Senti- WordNet (Esuli and Sebastiani, 2006) and a collec- tion of crawled tweets, in addition to unigrams or bigrams in the target tweet. For the Neural Net- work approach, we implement CNN based on Kim (2014). As the input embeddings, we use word embeddings trained by Continuous Bag-Of-Words (CBOW) model (Mikolov et al., 2013) on Wikipedia articles. The experimental results show that the CNN based approach performed the best in the cross val- idation on the training data. However the tendency was opposite on the test data probably because the CNN model overfitted to the training data. In con- trast, the feature-based approach was more robust, leveraging the external knowledge. 2 Datasets We use the dataset of the SemEval-2016 Task 6 Sub- task A, which is a supervised tweet classification task for five topics. There are three stances to clas- sify; NONE, FAVOR, AGAINST. Table 1 shows the topics and distributions of the training data. 401 Topic FAVOR AGAINST NONE Atheism 92 304 117 Climate Change is a Real Concern 212 15 168 Feminist Movement 210 328 126 Hillary Clinton 112 361 166 Legalization of Abortion …

Content truncated to 500 words