Neural Tensor Networks with Diagonal Slice Matrices

Proceedings of NAACL-HLT 2018, pages 506–515 New Orleans, Louisiana, June 1 - 6, 2018. c⃝2018 Association for Computational Linguistics Neural Tensor Networks with Diagonal Slice Matrices Takahiro Ishihara1 Katsuhiko Hayashi2 Hitoshi Manabe1 Masahi Shimbo1 Masaaki Nagata3 1 Nara Institute of Science and Technology 2 Osaka University 3 NTT Communication Science Laboratories 1 {ishihara.takahiro.in0, manabe.hitoshi.me0, shimbo}@is.naist.jp 2 khayashi0201@gmail.com 3 nagata.masaaki@lab.ntt.co.jp Abstract Although neural tensor networks (NTNs) have been successful in many natural language pro- cessing tasks, they require a large number of parameters to be estimated, which often results in overfitting and long training times. We ad- dress these issues by applying eigendecompo- sition to each slice matrix of a tensor to reduce the number of parameters. We evaluate our proposed NTN models in two tasks. First, the proposed models are evaluated in a knowledge graph completion task. Second, a recursive NTN (RNTN) extension of the proposed mod- els is evaluated on a logical reasoning task. The experimental results show that our pro- posed models learn better and faster than the original (R)NTNs. 1 Introduction Alongside the nonlinear activation functions, lin- ear mapping by matrix multiplication is an es- sential component of neural network (NN) mod- els, as it determines the feature interaction and thus the expressiveness of models. In addition to the matrix-based mapping, neural tensor net- works (NTNs) (Socher et al., 2013a) employ a 3- dimensional tensor to capture direct interactions among input features. Due to the large expres- sive capacity of 3D tensors, NTNs have been suc- cessful in an array of natural language process- ing (NLP) and machine learning tasks, includ- ing knowledge graph completion (KGC) (Socher et al., 2013a), sentiment analysis (Socher et al., 2013b), and reasoning with logical semantics (Bowman et al., 2015). However, since a 3D ten- sor has a large number of parameters, NTNs need longer time to train than other NN models. More- over, the millions of parameters often make the model suffer from overfitting (Yang et al., 2015). To solve these problems, we propose two new parameter reduction techniques for NTNs. These techniques drastically decrease the number of pa- rameters in an NTN without diminishing its ex- pressiveness. We use the matrix decomposition techniques that are utilized for KGC in Yang et al. (2015) and Trouillon et al. (2016). Yang et al. (2015) imposed a constraint that a matrix in the bilinear term in their model had to be diagonal. As mentioned in a subsequent section, this is es- sentially equal to assuming that the matrix be symmetric and performing eigendecomposition. Trouillon et al. (2016) also applied eigendecom- position to a matrix by regarding it as the real part of a normal matrix. Following these studies, we perform simultaneous diagonalization on all slice matrices of a NTN tensor. As a result, mapping by a 3D (n × n × k) tensor is replaced with an array of k “triple inner products” of two input vec- tors and a weight vector. Thus, we obtain two new NTN models where the number of …

Content truncated to 500 words