Simple Customization of Recursive Neural Networks for Semantic Relation Classification

Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1372–1376, Seattle, Washington, USA, 18-21 October 2013. c⃝2013 Association for Computational Linguistics Simple Customization of Recursive Neural Networks for Semantic Relation Classification Kazuma Hashimoto†, Makoto Miwa††, Yoshimasa Tsuruoka†, and Takashi Chikayama† †The University of Tokyo, 3-7-1 Hongo, Bunkyo-ku, Tokyo, Japan {hassy, tsuruoka, chikayama}@logos.t.u-tokyo.ac.jp ††The University of Manchester, 131 Princess Street, Manchester, M1 7DN, UK makoto.miwa@manchester.ac.uk Abstract In this paper, we present a recursive neural network (RNN) model that works on a syn- tactic tree. Our model differs from previous RNN models in that the model allows for an explicit weighting of important phrases for the target task. We also propose to average param- eters in training. Our experimental results on semantic relation classification show that both phrase categories and task-specific weighting significantly improve the prediction accuracy of the model. We also show that averaging the model parameters is effective in stabilizing the learning and improves generalization capacity. The proposed model marks scores competitive with state-of-the-art RNN-based models. 1 Introduction Recursive Neural Network (RNN) models are promising deep learning models which have been applied to a variety of natural language processing (NLP) tasks, such as sentiment classification, com- pound similarity, relation classification and syntactic parsing (Hermann and Blunsom, 2013; Socher et al., 2012; Socher et al., 2013). RNN models can repre- sent phrases of arbitrary length in a vector space of a fixed dimension. Most of them use minimal syn- tactic information (Socher et al., 2012). Recently, Hermann and Blunsom (2013) pro- posed a method for leveraging syntactic informa- tion, namely CCG combinatory operators, to guide composition of phrases in RNN models. While their models were successfully applied to binary senti- ment classification and compound similarity tasks, there are questions yet to be answered, e.g., whether such enhancement is beneficial in other NLP tasks as well, and whether a similar improvement can be achieved by using syntactic information of more commonly available types such as phrase categories and syntactic heads. In this paper, we present a supervised RNN model for a semantic relation classification task. Our model is different from existing RNN models in that impor- tant phrases can be explicitly weighted for the task. Syntactic information used in our model includes part-of-speech (POS) tags, phrase categories and syntactic heads. POS tags are used to assign vec- tor representations to word-POS pairs. Phrase cate- gories are used to determine which weight matrices are chosen to combine phrases. Syntactic heads are used to determine which phrase is weighted during combining phrases. To incorporate task-specific in- formation, phrases on the path between entity pairs are further weighted. The second contribution of our work is the intro- duction of parameter averaging into RNN models. In our preliminary experiments, we observed that the prediction performance of the model often fluc- tuates significantly between training iterations. This fluctuation not only leads to unstable performance of the resulting models, but also makes it difficult to fine-tune the hyperparameters of the model. Inspired by Swersky …

Content truncated to 500 words