Feature-guided Neural Model Training for Supervised Document Representation Learning

Feature-guided Neural Model Training for Supervised Document Representation Learning Aili Shen Bahar Salehi Jianzhong Qi Timothy Baldwin School of Computing and Information Systems The University of Melbourne Victoria, Australia ailis@student.unimelb.edu.au jianzhong.qi@unimelb.edu.aubaharsalehi@gmail.com tb@ldwin.net Abstract With the advent of neural models, there has been a rapid move away from feature engineer- ing, or at best, simplistically combining hand- crafted features with learned representations as side information. We propose a method that uses hand-crafted features to guide learn- ing by explicitly attending to feature indica- tors when learning the relationship between the input and target variables. In experiments over two different tasks — quality assessment of Wikipedia articles and popularity predic- tion of online petitions— we demonstrate that the proposed method yields neural models that consistently outperform those that simply use hand-crafted features as side information. 1 Introduction and Background Text classification/regression is a fundamental problem in natural language processing. Tradi- tional methods make use of hand-crafted features, such as the length of a document, to represent a document. A classifier/regressor is built on top of such features to learn a model (Wang and Man- ning, 2012; Warncke-Wang et al., 2013, 2015; Dang and Ignat, 2016). Recently, neural mod- els such as LSTMs (Hochreiter and Schmidhuber, 1997) and convolutional neural networks (CNNs: Kim (2014); Kalchbrenner et al. (2014)) have be- come the de facto for text classification/regression tasks, with one oft-cited advantage being that they are able to learn implicit features as part of the rep- resentation learning. Studies employing neural models either eschew hand-crafted features or simplistically use hand- crafted features as side information. For example, Dang and Ignat (2017) propose to use a bidirec- tional LSTM (“bi-LSTM”) to classify Wikipedia articles by their quality classes, and Shen et al. (2017) concatenate structural features (e.g., arti- cle length) and readability scores with bi-LSTM- learned document representations for the same task. Subramanian et al. (2018) hand-engineer a set of features (e.g., the ratio of indefinite and def- inite articles), and concatenate them with CNN- learned document representations to predict the popularity of online petitions. Wu et al. (2018) explore the utility of hand-crafted features in NER by concatenating these features with character rep- resentations learned via an CNN and word embed- dings. These representations are then fed into a bi- LSTM to identify named entities (with the help of a CRF) and re-construct the hand-crafted features in the output simultaneously, which is achieved by combining an auto-encoder loss with the NER loss. The motivation underlying this work is that when hand-crafted features are represented by nu- merical vectors and concatenated with neural net- work representations, there is no information on what kind of feature each value represents. To make better use of hand-crafted features, we pro- pose a feature-guided neural training method that guides the network to map feature indicators onto (explicit or implicit) features in the document. We evaluate the effectiveness of the proposed method over two datasets for two different tasks: (1) qual- ity assessment of Wikipedia articles, and (2) pop- …

Content truncated to 500 words