Representing Syntax and Composition with Geometric Transformations

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 3343–3353 August 1–6, 2021. ©2021 Association for Computational Linguistics 3343 Representing Syntax and Composition with Geometric Transformations Lorenzo Bertolini Julie Weeds David Weir Qiwei Peng University of Sussex Brighton, UK {l.bertolini, j.e.weeds, d.j.weir, qiwei.peng}@sussex.ac.uk Abstract The exploitation of syntactic graphs (SyGs) as a word’s context has been shown to be beneficial for distributional semantic models (DSMs), both at the level of individual word representations and in deriving phrasal repre- sentations via composition. However, notwith- standing the potential performance benefit, the syntactically-aware DSMs proposed to date have huge numbers of parameters (compared to conventional DSMs) and suffer from data sparsity. Furthermore, the encoding of the SyG links (i.e., the syntactic relations) has been largely limited to linear maps. The knowledge graphs’ literature, on the other hand, has proposed light-weight models em- ploying different geometric transformations (GTs) to encode edges in a knowledge graph (KG). Our work explores the possibility of adopting this family of models to encode SyGs. Furthermore, we investigate which GT better encodes syntactic relations, so that these repre- sentations can be used to enhance phrase-level composition via syntactic contextualisation. 1 Introduction Representing words in terms of their syntactic co-occurrences has been long proposed, both for count-based (Pad´o and Lapata, 2007; Weir et al., 2016), and neural (Hermann and Blunsom, 2013; Levy and Goldberg, 2014; Komninos and Man- andhar, 2016; Czarnowska et al., 2019; Vashishth et al., 2019) models of word meaning. Tested on benchmark word similarity tasks, such models of- ten perform favourably to models based on proxi- mal co-occurrence, particularly when the similar- ity or substitutability of two words is considered rather than their relatedness (Levy and Goldberg, 2014). However, the real promise of distributional models based on syntactic rather than proximal co-occurrence, is the potential for carrying out syntax-sensitive composition. For example, in the Anchored Packed Tree (APT) model (Weir et al., 2016) lexemes, phrases, and sentences are repre- sented as collections of typed occurrences, and composition is carried out by contextualising each element in its syntactic role. This leads to syntax- sensitive representations for phrases. For exam- ple, glass window and window glass have different representations due to the different syntactic roles played by each constituent. Alongside count-based models, a variety of neural ones have been proposed to encode syn- tactic structure, focusing on different depths of the graph (Levy and Goldberg, 2014; Komninos and Manandhar, 2016; Marcheggiani and Titov, 2017; Vashishth et al., 2019; Emerson, 2020)). Of particular note here, Levy and Goldberg (2014) and Komninos and Manandhar (2016) each proposed models (DEP and EXT, respectively) which learn from local dependency relations, by extending the Skip-Gram with Negative sampling (SGNS) architecture from word2vec (Mikolov et al., 2013). Given a tuple of (target, context) words, e.g. (rain,like), a standard SGNS model can be trained to encode the probability of it being a true or a randomly sampled tuple. DEP and EXT, on the other hand, make use of both standard and syntactically contextualised tuples e.g., …

Content truncated to 500 words