Deep LSTM based Feature Mapping for Query Classification

Proceedings of NAACL-HLT 2016, pages 1501–1511, San Diego, California, June 12-17, 2016. c⃝2016 Association for Computational Linguistics Deep LSTM based Feature Mapping for Query Classification Yangyang Shi and Kaisheng Yao and Le Tian and Daxin Jiang Microsoft {yangshi,kaisheng.yao,letian,djiang}@microsoft.com Abstract Traditional convolutional neural network (CNN) based query classification uses linear feature mapping in its convolution opera- tion. The recurrent neural network (RNN), differs from a CNN in representing word sequence with their ordering information kept explicitly. We propose using a deep long-short-term-memory (DLSTM) based fea- ture mapping to learn feature representation for CNN. The DLSTM, which is a stack of LSTM units, has different order of feature representations at different depth of LSTM unit. The bottom LSTM unit equipped with input and output gates, extracts the first order feature representation from current word. To extract higher order nonlinear feature representation, the LSTM unit at higher position gets input from two parts. First part is the lower LSTM unit’s memory cell from previous word. Second part is the lower LSTM unit’s hidden output from current word. In this way, the DLSTM captures the nonlinear nonconsecutive interaction within n-grams. Using an architecture that combines a stack of the DLSTM layers with a tradition CNN layer, we have observed new state-of-the-art query classification accuracy on benchmark data sets for query classification. 1 Introduction Convolutional neural networks (CNNs) have achieved significant improvements for query classi- fication. CNNs capture the correlations of spatial or temporal structures with different resolutions using their temporal convolution operators. A pooling strategy on these local correlations extracts invariant regularities. However, CNNs use simple linear operations on n- gram vectors that are formed by concatenating word vectors. The linear operation together with the con- catenation may not be sufficient to model the non- consecutive dependency and interaction within the n-grams. For example, in the query “not a total loss”, nonconsecutive dependency “not loss” is the key information that is not well addressed by the lin- ear operation with simple concatenation. In this paper, we propose to use deep long-short- term-memory (DLSTM) based feature mapping to capture high order nonlinear feature representations. LSTM (Hochreiter and Schmidhuber, 1997) is one type of recurrent neural networks (RNNs) that have achieved remarkable performance in natural lan- guage processing and speech recognition (Sutskever et al., 2014; Graves et al., 2013). The DLSTM is a stack of LSTM units where dif- ferent order of nonlinear feature representation is captured by LSTM units at different depth. The bot- tom LSTM unit extracts the first order feature repre- sentation from current word. The LSTM unit at the higher position captures the higher order feature rep- resentation relying on the outputs from LSTM units at lower position, specifically, the memory cell from lower LSTM unit at previous word position and the hidden output from lower LSTM unit at current word position. Using DLSTM, linear feature mapping in traditional CNN can be obviously extended to non- linear feature mapping. Moreover, the memory cell together with different gates in LSTM unit are able …

Content truncated to 500 words