Improved Neural Network-based Multi-label Classification with Better Initialization Leveraging Label Co-occurrence

Proceedings of NAACL-HLT 2016, pages 521–526, San Diego, California, June 12-17, 2016. c⃝2016 Association for Computational Linguistics Improved Neural Network-based Multi-label Classification with Better Initialization Leveraging Label Co-occurrence Gakuto Kurata IBM Research gakuto@jp.ibm.com Bing Xiang IBM Watson bingxia@us.ibm.com Bowen Zhou IBM Watson zhou@us.ibm.com Abstract In a multi-label text classification task, in which multiple labels can be assigned to one text, label co-occurrence itself is informative. We propose a novel neural network initializa- tion method to treat some of the neurons in the final hidden layer as dedicated neurons for each pattern of label co-occurrence. These dedicated neurons are initialized to connect to the corresponding co-occurring labels with stronger weights than to others. In experi- ments with a natural language query classifi- cation task, which requires multi-label clas- sification, our initialization method improved classification accuracy without any computa- tional overhead in training and evaluation. 1 Introduction In multi-label text classification, one text can be associated with multiple labels (label co- occurrence) (Zhang and Zhou, 2014). Since la- bel co-occurrence itself contains information, we would like to leverage the label co-occurrence to im- prove multi-label classification using a neural net- work (NN). We propose a novel NN initialization method that treats some of the neurons in the final hidden layer as dedicated neurons for each pattern of label co-occurrence. These dedicated neurons are initialized to connect to the corresponding co- occurring labels with stronger weights than to oth- ers. While initialization of an NN is an important research topic (Glorot and Bengio, 2010; Sutskever et al., 2013; Le et al., 2015), to the best of our knowl- edge, there has been no attempt to leverage label co- occurrence for NN initialization. To validate our proposed method, we focus on multi-label Natural Language Query (NLQ) classifi- cation in a document retrieval system in which users input queries in natural language and the system re- turns documents that contain answers to the queries. For NLQ classification, we first train a model from training data that contains pairs of queries and cor- responding one or more than one document labels, and then predict the appropriate document labels for new queries with the trained model. Through experiments with a real-world document retrieval system and publicly available multi-label data set, simply and directly embedding label co- occurrence information into an NN with our pro- posed method improved accuracy of NLQ classifi- cation. 2 Related Work Along with the recent success in NNs (Collobert et al., 2011; Kim, 2014), NN-based multi-label classi- fication has been proposed. An NN for NLQ classi- fication needs to accept queries with variable length and output their labels. Figure 1 shows a typical NN architecture (Collobert et al., 2011). This NN first transforms words in the input query into word em- beddings (Mikolov et al., 2013), then applies Con- volutional Neural Network (CNN) and Max-pooling over time to extract fixed-length feature vectors, and feed them into the output layer to predict the label for the query (Collobert and Weston, 2008; Col- lobert et …

Content truncated to 500 words