Reduction of Parameter Redundancy in Biaffine Classifiers with Symmetric and Circulant Weight Matrices

Reduction of Parameter Redundancy in Biaffine Classifiers with Symmetric and Circulant Weight Matrices Tomoki Matsuno1, Katsuhiko Hayashi2,4, Takahiro Ishihara1, Hitoshi Manabe3, Yuji Matsumoto1,4 1Nara Institute of Science and Technology 2Osaka University 3Works Applications 4RIKEN Center for Advanced Intelligence Project 1{matsuno.tomoki.mr1,ishihara.takahiro.in0, matsu}@is.naist.jp 2khayashi0201@gmail.com 3manabe h@worksap.co.jp Abstract Currently, the biaffine classifier has been at- tracting attention as a method to introduce an attention mechanism into the modeling of bi- nary relations. For instance, in the field of dependency parsing, the Deep Biaffine Parser by Dozat and Manning has achieved state- of-the-art performance as a graph-based de- pendency parser on the English Penn Tree- bank and CoNLL 2017 shared task. On the other hand, it is reported that parameter redun- dancy in the weight matrix in biaffine classi- fiers, which has O(n2) parameters, results in overfiting (n is the number of dimensions). In this paper, we attempted to reduce the param- eter redundancy by assuming either symme- try or circularity of weight matrices. In our experiments on the CoNLL 2017 shared task dataset, our model achieved better or compa- rable accuracy on most of the treebanks with more than 16% parameter reduction. 1 Introduction Recently, methods based on attention mechanisms have been used in various fields of natural lan- guage processing (Bahdanau et al., 2014; Rush et al., 2015). In tasks which deal with binary relations that take two words as arguments, e.g., depen- dency parsing, a good number of these models have achieved high performance (Kiperwasser and Gold- berg, 2016; Hashimoto et al., 2017; Dozat and Man- ning, 2016). Biaffine transformation is a method to incor- porate an attention mechanism into binary rela- tions proposed by Dozat and Manning (2016) (fol- lowing them, we call this method biaffine clas- sifier). They achieved the state-of-the-art perfor- mance among graph-based dependency parsers for the English Penn Treebank. In addition, the state-of- the-art transition-based parser on English Penn Tree- bank uses a biaffine classifier to evaluate the proba- bility distribution of a word coming into the stack at a step point (Ma et al., 2018). While biaffine transformation has rich expressive- ness in modeling binary relations, its number of parameters in the weight matrix (bilinear term) is O(n2) (where n is the number of dimensions). This redundant parameters can result in high degree of freedom of the model, thus causing overfitting espe- cially when a large number of training samples are not available (Nickel et al., 2015). In this paper, we attempt to reduce the redundancy by introducing the assumption of either symmetry or circularity in the weight matrix at a biaffine classi- fier. With either assumption, we can vectorize the matrix and reduce the space complexity to O(n). Additionally, the time complexity becomes O(n) in the case of symmetry and O(n log n) in the case of circularity with the fast Fourier transform. Further- more, while the expressiveness of the model based on symmetry is restricted1, one based on circularity is able to express asymmetry relations. In our experiments, we imposed constraints on the …

Content truncated to 500 words