Assessing Composition in Sentence Vector Representations

Proceedings of the 27th International Conference on Computational Linguistics, pages 1790–1801 Santa Fe, New Mexico, USA, August 20-26, 2018. 1790 Assessing Composition in Sentence Vector Representations Allyson Ettinger1, Ahmed Elgohary2, Colin Phillips1, Philip Resnik1,3 1Linguistics, 2Computer Science, 3Institute for Advanced Computer Studies University of Maryland, College Park, MD {aetting,colin,resnik}@umd.edu, elgohary@cs.umd.edu Abstract An important component of achieving language understanding is mastering the composition of sentence meaning, but an immediate challenge to solving this problem is the opacity of sentence vector representations produced by current neural sentence composition models. We present a method to address this challenge, developing tasks that directly target compositional meaning information in sentence vector representations with a high degree of precision and control. To enable the creation of these controlled tasks, we introduce a specialized sentence generation sys- tem that produces large, annotated sentence sets meeting specified syntactic, semantic and lexical constraints. We describe the details of the method and generation system, and then present results of experiments applying our method to probe for compositional information in embeddings from a number of existing sentence composition models. We find that the method is able to extract use- ful information about the differing capacities of these models, and we discuss the implications of our results with respect to these systems’ capturing of sentence information. We make available for public use the datasets used for these experiments, as well as the generation system.1 1 Introduction As natural language processing strives toward language understanding, it is important that we develop models able to extract and represent the meaning of sentences. Such representations promise to be applicable across a variety of tasks, and to be more robust than non-meaning-based representations for any given task requiring meaning understanding. To accomplish meaning extraction, a particular need is that of mastering composition: systematic derivation of the meaning of a sentence based on its parts. In this paper we tackle compositional meaning extraction by first addressing the challenge of evalua- tion and interpretability: after all, in order to improve meaning extraction, we need to be able to evaluate it. But with sentence representations increasingly taking the form of dense vectors (embeddings) from neural network models, it is difficult to assess what information these representations are capturing—and this problem is particularly acute for assessing abstract content like compositional meaning information. Here we introduce an analysis method for targeting and evaluating compositional meaning informa- tion in sentence embeddings. The approach builds on a proposal outlined in Ettinger et al. (2016), and involves designing classification tasks that directly target the information of interest (e.g., “Given a noun n, verb v, and an embedding s of sentence s: is n the agent of v in s?”). By contrast to re- lated work analyzing surface variables like word content and word order in sentence embeddings (Adi et al., 2016), we specifically target compositional meaning information relevant to achieving language understanding—and in order to isolate this more abstract information, we exert careful control over our classification datasets to ensure that we are targeting …

Content truncated to 500 words