Autoencoding Pixies: Amortised Variational Inference with Graph Convolutions for Functional Distributional Semantics

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3982–3995 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 3982 Autoencoding Pixies: Amortised Variational Inference with Graph Convolutions for Functional Distributional Semantics Guy Emerson Department of Computer Science and Technology University of Cambridge gete2@cam.ac.uk Abstract Functional Distributional Semantics provides a linguistically interpretable framework for distributional semantics, by representing the meaning of a word as a function (a binary clas- sifier), instead of a vector. However, the large number of latent variables means that infer- ence is computationally expensive, and train- ing a model is therefore slow to converge. In this paper, I introduce the Pixie Autoencoder, which augments the generative model of Func- tional Distributional Semantics with a graph- convolutional neural network to perform amor- tised variational inference. This allows the model to be trained more effectively, achiev- ing better results on two tasks (semantic sim- ilarity in context and semantic composition), and outperforming BERT, a large pre-trained language model. 1 Introduction The aim of distributional semantics is to learn the meanings of words from a corpus (Harris, 1954; Firth, 1951, 1957). Many approaches learn a vector for each word, including count models and em- bedding models (for an overview, see: Erk, 2012; Clark, 2015), and some recent approaches learn a vector for each token in a particular context (for example: Peters et al., 2018; Devlin et al., 2019). However, such vector representations do not make a clear distinction between words and the things they refer to. This means that such models are challenging to interpret semantically. In con- trast, Functional Distributional Semantics (Emer- son and Copestake, 2016) aims to provide a frame- work which can be interpreted in terms of model theory, a standard approach to formal semantics. Furthermore, this framework supports first-order logic, where quantifying over logical variables is replaced by marginalising out random variables (Emerson and Copestake, 2017b; Emerson, 2020b). This connection to logic is a clear strength over vector-based models. Even the linguistically in- spired tensor-based framework of Coecke et al. (2010) and Baroni et al. (2014) cannot model quan- tifiers, as shown by Grefenstette (2013). However, the linguistic interpretability of Func- tional Distributional Semantics comes at a compu- tational cost, with a high-dimensional latent vari- able for each token. Training a model by gradient descent requires performing Bayesian inference over these latent variables, which is intractable to calculate exactly. The main theoretical contribution of this paper is to present an amortised variational inference algorithm to infer these latent variables. This is done using a graph-convolutional network, as described in §3. The main empirical contribution of this paper is to demonstrate that the resulting system, the Pixie Autoencoder, improves performance on two seman- tic tasks, as described in §4. I also present the first published results of applying a large language model (BERT) to these tasks, showing that results are sensitive to linguistic detail in how the model is applied. Despite being a smaller model trained on less data, the Pixie …

Content truncated to 500 words