Hierarchical Attention Networks for Document Classification

Proceedings of NAACL-HLT 2016, pages 1480–1489, San Diego, California, June 12-17, 2016. c⃝2016 Association for Computational Linguistics Hierarchical Attention Networks for Document Classification Zichao Yang1, Diyi Yang1, Chris Dyer1, Xiaodong He2, Alex Smola1, Eduard Hovy1 1Carnegie Mellon University, 2Microsoft Research, Redmond {zichaoy, diyiy, cdyer, hovy}@cs.cmu.edu xiaohe@microsoft.com alex@smola.org Abstract We propose a hierarchical attention network for document classification. Our model has two distinctive characteristics: (i) it has a hier- archical structure that mirrors the hierarchical structure of documents; (ii) it has two levels of attention mechanisms applied at the word- and sentence-level, enabling it to attend dif- ferentially to more and less important con- tent when constructing the document repre- sentation. Experiments conducted on six large scale text classification tasks demonstrate that the proposed architecture outperform previous methods by a substantial margin. Visualiza- tion of the attention layers illustrates that the model selects qualitatively informative words and sentences. 1 Introduction Text classification is one of the fundamental task in Natural Language Processing. The goal is to as- sign labels to text. It has broad applications includ- ing topic labeling (Wang and Manning, 2012), senti- ment classification (Maas et al., 2011; Pang and Lee, 2008), and spam detection (Sahami et al., 1998). Traditional approaches of text classification repre- sent documents with sparse lexical features, such as n-grams, and then use a linear model or kernel methods on this representation (Wang and Manning, 2012; Joachims, 1998). More recent approaches used deep learning, such as convolutional neural net- works (Blunsom et al., 2014) and recurrent neural networks based on long short-term memory (LSTM) (Hochreiter and Schmidhuber, 1997) to learn text representations. pork belly = delicious . || scallops? || I don’t even like scallops, and these were a-m-a-z-i-n-g . || fun and tasty cocktails. || next time I in Phoenix, I will go back here. || Highly recommend. Figure 1: A simple example review from Yelp 2013 that con- sists of five sentences, delimited by period, question mark. The first and third sentence delivers stronger meaning and inside, the word delicious, a-m-a-z-i-n-g contributes the most in defin- ing sentiment of the two sentences. Although neural-network–based approaches to text classification have been quite effective (Kim, 2014; Zhang et al., 2015; Johnson and Zhang, 2014; Tang et al., 2015), in this paper we test the hypoth- esis that better representations can be obtained by incorporating knowledge of document structure in the model architecture. The intuition underlying our model is that not all parts of a document are equally relevant for answering a query and that determining the relevant sections involves modeling the interac- tions of the words, not just their presence in isola- tion. Our primary contribution is a new neural archi- tecture (§2), the Hierarchical Attention Network (HAN) that is designed to capture two basic insights about document structure. First, since documents have a hierarchical structure (words form sentences, sentences form a document), we likewise construct a document representation by first building represen- tations of sentences and then aggregating those into a document representation. Second, …

Content truncated to 500 words