Learning Tag Embeddings and Tag-specific Composition Functions in Recursive Neural Network

Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing, pages 1365–1374, Beijing, China, July 26-31, 2015. c⃝2015 Association for Computational Linguistics Learning Tag Embeddings and Tag-specific Composition Functions in Recursive Neural Network Qiao Qian, Bo Tian, Minlie Huang, Yang Liu*, Xuan Zhu*, Xiaoyan Zhu State Key Lab. of Intelligent Technology and Systems, National Lab. for Information Science and Technology, Dept. of Computer Science and Technology, Tsinghua University, Beijing 100084, PR China *Samsung R&D Institute Beijing, China qianqiaodecember29@126.com , smxtianbo@gmail.com aihuang@tsinghua.edu.cn , yang.liu@samsung.com xuan.zhu@samsung.com , zxy-dcs@tsinghua.edu.cn Abstract Recursive neural network is one of the most successful deep learning models for natural language processing due to the compositional nature of text. The model recursively composes the vector of a parent phrase from those of child words or phrases, with a key compo- nent named composition function. Al- though a variety of composition func- tions have been proposed, the syntactic information has not been fully encoded in the composition process. We pro- pose two models, Tag Guided RNN (TG- RNN for short) which chooses a compo- sition function according to the part-of- speech tag of a phrase, and Tag Embedded RNN/RNTN (TE-RNN/RNTN for short) which learns tag embeddings and then combines tag and word embeddings to- gether. In the fine-grained sentiment classification, experiment results show the proposed models obtain remarkable improvement: TG-RNN/TE-RNN obtain remarkable improvement over baselines, TE-RNTN obtains the second best result among all the top performing models, and all the proposed models have much less parameters/complexity than their counter- parts. 1 Introduction Among a variety of deep learning models for nat- ural language processing, Recursive Neural Net- work (RNN) may be one of the most popular mod- els. Thanks to the compositional nature of natu- ral text, recursive neural network utilizes the re- cursive structure of the input such as a phrase or sentence, and has shown to be very effective for many natural language processing tasks including semantic relationship classification (Socher et al., 2012), syntactic parsing (Socher et al., 2013a), sentiment analysis (Socher et al., 2013b), and ma- chine translation (Li et al., 2013). The key component of RNN and its variants is the composition function: how to compose the vector representation for a longer text from the vector of its child words or phrases. For instance, as shown in Figure 2, the vector of ‘is very inter- esting’ can be composed from the vector of the left node ‘is’ and that of the right node ‘very interest- ing’. It’s worth to mention again, the composition process is conducted with the syntactic structure of the text, making RNN more interpretable than other deep learning models. so#max so#max so#max so#max so#max ... g g very interesting is very interesting is very interesting Figure 1: The example process of vector composi- tion in RNN. The vector of node ‘very interesting’ is composed from the vectors of node ‘very’ and node ‘interesting’. Similarly, the node ‘is very in- teresting’ is …

Content truncated to 500 words