Adapting RNN Sequence Prediction Model to Multi-label Set Prediction

Proceedings of NAACL-HLT 2019, pages 3181–3190 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 3181 Adapting RNN Sequence Prediction Model to Multi-label Set Prediction Kechen Qin Cheng Li Virgil Pavlu Javed A. Aslam Khoury College of Computer Sciences Northeastern University qin.ke@husky.neu.edu {chengli,vip,jaa}@ccs.neu.edu Abstract We present an adaptation of RNN sequence models to the problem of multi-label classifi- cation for text, where the target is a set of la- bels, not a sequence. Previous such RNN mod- els define probabilities for sequences but not for sets; attempts to obtain a set probability are after-thoughts of the network design, includ- ing pre-specifying the label order, or relating the sequence probability to the set probability in ad hoc ways. Our formulation is derived from a princi- pled notion of set probability, as the sum of probabilities of corresponding permutation se- quences for the set. We provide a new training objective that maximizes this set probability, and a new prediction objective that finds the most probable set on a test document. These new objectives are theoretically appealing be- cause they give the RNN model freedom to discover the best label order, which often is the natural one (but different among documents). We develop efficient procedures to tackle the computation difficulties involved in training and prediction. Experiments on benchmark datasets demonstrate that we outperform state- of-the-art methods for this task. 1 Introduction Multi-label text classification is an important ma- chine learning task wherein one must predict a set of labels to associate with a given document; for example, a news article might be tagged with labels sport, football, 2018 world cup, and Russia. Formally, we are given a set of label candidates L = {1, 2, ..., L}, and we aim to build a classifier which maps a document x to a set of labels y ⊂ L. The label set y is typically written as a binary vector y ∈ {0, 1}L, with each bit yℓ indicating the presence or absence of a label. Naively, one could predict each label indepen- dently without considering label dependencies. This approach is called Binary Relevance (Boutell et al., 2004; Tsoumakas and Katakis, 2007), and is widely used due to its simplicity, but it of- ten does not deliver good performance. Intu- itively, knowing some labels—such as sport and football—should make it easier to predict 2018 world cup and then Russia. There are several methods that try to capture label de- pendencies by building a joint probability esti- mation over all labels p(y = (y1, y2, ..., yL)|x) (Ghamrawi and McCallum, 2005; Read et al., 2009; Dembczynski et al., 2010; Li et al., 2016). The most popular approach, Probabilistic Clas- sifier Chain (PCC) (Dembczynski et al., 2010) learns labels one-by-one in a predefined fixed or- der: for each label, it uses one classifier to estimate the probability of that label given all previous la- bels predictions, p(yl|y1, ..., yl−1, x). PCC’s well known drawback is that errors in early probability estimations tend to …

Content truncated to 500 words