Reducing Dimensions of Tensors in Type-Driven Distributional Semantics

Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1036–1046, October 25-29, 2014, Doha, Qatar. c⃝2014 Association for Computational Linguistics Reducing Dimensions of Tensors in Type-Driven Distributional Semantics Tamara Polajnar Luana Fˇagˇarˇas¸an Stephen Clark Computer Laboratory University of Cambridge Cambridge, UK first.last@cl.cam.ac.uk Abstract Compositional distributional semantics is a subfield of Computational Linguistics which investigates methods for represent- ing the meanings of phrases and sen- tences. In this paper, we explore im- plementations of a framework based on Combinatory Categorial Grammar (CCG), in which words with certain grammatical types have meanings represented by multi- linear maps (i.e. multi-dimensional arrays, or tensors). An obstacle to full implemen- tation of the framework is the size of these tensors. We examine the performance of lower dimensional approximations of tran- sitive verb tensors on a sentence plausi- bility/selectional preference task. We find that the matrices perform as well as, and sometimes even better than, full tensors, allowing a reduction in the number of pa- rameters needed to model the framework. 1 Introduction An emerging subfield of computational linguis- tics is concerned with learning compositional dis- tributional representations of meaning (Mitchell and Lapata, 2008; Baroni and Zamparelli, 2010; Coecke et al., 2010; Grefenstette and Sadrzadeh, 2011; Clarke, 2012; Socher et al., 2012; Clark, 2013). The advantage of such representations lies in their potential to combine the benefits of dis- tributional approachs to word meaning (Sch¨utze, 1998; Turney and Pantel, 2010) with the more tra- ditional compositional methods from formal se- mantics (Dowty et al., 1981). Distributional repre- sentations have the properties of robustness, learn- ability from data, ease of handling ambiguity, and the ability to represent gradations of mean- ing; whereas compositional models handle the un- bounded nature of natural language, as well as providing established accounts of logical words, quantification, and inference. One promising approach which attempts to combine elements of compositional and distribu- tional semantics is by Coecke et al. (2010). The underlying idea is to take the type-driven approach from formal semantics — in particular the idea that the meanings of complex grammatical types should be represented as functions — and ap- ply it to distributional representations. Since the mathematics of distributional semantics is pro- vided by linear algebra, a natural set of functions to consider is the set of linear maps. Coecke et al. recognize that there is a natural correspon- dence from complex grammatical types to tensors (multi-linear maps), so that the meaning of an ad- jective, for example, is represented by a matrix (a 2nd-order tensor)1 and the meaning of a transitive verb is represented by a 3rd-order tensor. Coecke et al. use the grammar of pregroups as the syntactic machinery to construct distribu- tional meaning representations, since both pre- groups and vector spaces can be seen as exam- ples of the same abstract structure, which leads to a particularly clean mathematical description of the compositional process. However, the approach applies more generally, for example to other forms of categorial grammar, such as Combinatory Cate- …

Content truncated to 500 words