Leveraging Preposition Ambiguity to Assess Compositional Distributional Models of Semantics

Proceedings of the Fourth Joint Conference on Lexical and Computational Semantics (*SEM 2015), pages 199–204, Denver, Colorado, June 4–5, 2015. Leveraging Preposition Ambiguity to Assess Compositional Distributional Models of Semantics Samuel Ritter∗ Princeton University Cotie Long Indiana University Denis Paperno University of Trento Marco Baroni University of Trento Matthew Botvinick Princeton University Adele Goldberg Princeton University Abstract Complex interactions among the meanings of words are important factors in the function that maps word meanings to phrase meanings. Recently, compositional distributional seman- tics models (CDSM) have been designed with the goal of emulating these complex interac- tions; however, experimental results on the ef- fectiveness of CDSM have been difficult to in- terpret because the current metrics for assess- ing them do not control for the confound of lexical information. We present a new method for assessing the degree to which CDSM cap- ture semantic interactions that dissociates the influences of lexical and compositional infor- mation. We then provide a dataset for per- forming this type of assessment and use it to evaluate six compositional models using both co-occurrence based and neural language model input vectors. Results show that neural language input vectors are consistently supe- rior to co-occurrence based vectors, that sev- eral CDSM capture substantial compositional information, and that, surprisingly, vector ad- dition matches and is in many cases superior to purpose-built paramaterized models. 1 Introduction Consider the meanings of the following phrases: “red apple,” “red hair,” and “red state.” The meaning of the word “red” in each of these examples interacts with the meaning of the noun it modifies, applying ∗Please address correspondence to the first author at swrit- ter@princeton.edu a different color to the first two and a political af- filiation to the third. This is an example of a com- mon phenomenon in natural language in which the meaning of a whole expression is not derived from a simple concatenation of its parts, but is composed by interactions among their meanings. Cognitive and computer scientists have pointed out this complexity and proposed various models for accommodating it (Kintsch, 2001; Mitchell and La- pata, 2010; Socher et al., 2013). A dominant model- ing approach seeks to learn functions that combine word representations derived from the distributional structure of large natural language corpora (Deer- wester et al., 1990; Landauer and Dumais, 1997). Because the word representations to be combined and the compositional functions are generated based on the distributions of words in corpora, these mod- els have been dubbed compositional distributional semantic models, or CDSM (Marelli et al., 2014). CDSM produce fixed-dimensional vector represen- tations of arbitrary sentences and phrases, and the foundational principle of these models is, stated sim- ply, that semantically similar phrases should have vector representations that are close together in the vector space. 1.1 CDSM Assessment Past studies have tested how well CDSM adhere to this principle by comparing the vector similarity of pairs of sentences with similarity ratings given by humans. Many of these studies used datasets in which the amount of lexical overlap …

Content truncated to 500 words