Multi-Timescale Long Short-Term Memory Neural Network for Modelling Sentences and Documents

Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 2326–2335, Lisbon, Portugal, 17-21 September 2015. c⃝2015 Association for Computational Linguistics. Multi-Timescale Long Short-Term Memory Neural Network for Modelling Sentences and Documents Pengfei Liu, Xipeng Qiu∗, Xinchi Chen, Shiyu Wu, Xuanjing Huang Shanghai Key Laboratory of Intelligent Information Processing, Fudan University School of Computer Science, Fudan University 825 Zhangheng Road, Shanghai, China {pfliu14,xpqiu,xinchichen13,syu13,xjhuang}@fudan.edu.cn Abstract Neural network based methods have ob- tained great progress on a variety of nat- ural language processing tasks. However, it is still a challenge task to model long texts, such as sentences and documents. In this paper, we propose a multi-timescale long short-term memory (MT-LSTM) neu- ral network to model long texts. MT- LSTM partitions the hidden states of the standard LSTM into several groups. Each group is activated at different time peri- ods. Thus, MT-LSTM can model very long documents as well as short sentences. Experiments on four benchmark datasets show that our model outperforms the other neural models in text classification task. 1 Introduction Distributed representations of words have been widely used in many natural language process- ing (NLP) tasks (Collobert et al., 2011; Turian et al., 2010; Mikolov et al., 2013b; Bengio et al., 2003). Following this success, it is rising a sub- stantial interest to learn the distributed represen- tations of the continuous words, such as phrases, sentences, paragraphs and documents (Mitchell and Lapata, 2010; Socher et al., 2013; Mikolov et al., 2013b; Le and Mikolov, 2014; Kalchbren- ner et al., 2014). The primary role of these mod- els is to represent the variable-length sentence or document as a fixed-length vector. A good rep- resentation of the variable-length text should fully capture the semantics of natural language. Recently, the long short-term memory neural network (LSTM) (Hochreiter and Schmidhuber, 1997) has been applied successfully in many NLP tasks, such as spoken language understanding (Yao et al., 2014), sequence labeling (Chen et al., ∗Corresponding author 2015) and machine translation (Sutskever et al., 2014). LSTM is an extension of the recurrent neu- ral network (RNN) (Elman, 1990), which can cap- ture the long-term and short-term dependencies and is very suitable to model the variable-length texts. Besides, LSTM is also sensitive to word order and does not rely on the external syntactic structure as recursive neural network (Socher et al., 2013). However, when modeling long texts, such as documents, LSTM need to keep the useful features for a quite long period of time. The long- term dependencies need to be transmitted one-by- one along the sequence. Some important features could be lost in transmission process. Besides, the error signal is also back-propagated one-by- one through multiple time steps in the training phase with back-propagation through time (BPTT) (Werbos, 1990) algorithm. The learning efficiency could also be decreased for the long texts. For ex- ample, if a valuable feature occurs at the begin of a long document, we need to back-propagate the error through the whole document. In this paper, we propose a multi-timescale …

Content truncated to 500 words