Embedding Imputation with Grounded Language Information

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3356–3361 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 3356 Embedding Imputation with Grounded Language Information Ziyi Yang1, Chenguang Zhu2, Vin Sachidananda3, and Eric Darve1, 4 1Department of Mechanical Engineering, Stanford University 2Microsoft Speech and Dialogue Research Group 3Department of Electrical Engineering, Stanford University 4Institute for Computational and Mathematical Engineering, Stanford University {ziyi.yang, vsachi, darve}@stanford.edu, chezhu@microsoft.com Abstract Due to the ubiquitous use of embeddings as input representations for a wide range of natu- ral language tasks, imputation of embeddings for rare and unseen words is a critical problem in language processing. Embedding imputa- tion involves learning representations for rare or unseen words during the training of an em- bedding model, often in a post-hoc manner. In this paper, we propose an approach for embed- ding imputation which uses grounded infor- mation in the form of a knowledge graph. This is in contrast to existing approaches which typ- ically make use of vector space properties or subword information. We propose an online method to construct a graph from grounded information and design an algorithm to map from the resulting graphical structure to the space of the pre-trained embeddings. Finally, we evaluate our approach on a range of rare and unseen word tasks across various domains and show that our model can learn better repre- sentations. For example, on the Card-660 task our method improves Pearson’s and Spear- man’s correlation coefficients upon the state- of-the-art by 11% and 17.8% respectively us- ing GloVe embeddings. 1 Introduction Word embeddings (Mikolov et al., 2013; Penning- ton et al., 2014) are used pervasively in deep learn- ing for natural language processing. However, due to fixed vocabulary constraints in existing ap- proaches to training word embeddings, it is diffi- cult to learn representations for words which are rare or unseen during training. This is commonly referred to as the out-of-vocabulary (OOV) word problem. In the original embedding implementa- tions, a special OOV token is typically reserved for such words. However, this rudimentary ap- proach often detriments the performance of down- stream tasks which contain numerous rare or un- seen words. Recent works have proposed subword approaches (Zhao et al., 2018; Sennrich et al., 2015), which construct embeddings through the composition of characters or sentence pieces for OOV words. Vector space properties are also uti- lized to learn embeddings with small amounts of data (Bahdanau et al., 2017; Herbelot and Ba- roni, 2017). In this paper, we propose a novel approach, knowledge-graph-to-vector (KG2Vec), for the OOV word problem. KG2Vec makes use of the grounded language information in the form of a knowledge graph. Grounded information has been extensively used in various NLP tasks to represent real-world knowledge (Niles and Pease, 2003; Gruber, 1993; Guarino, 1998; de Bruijn et al., 2006; Paulheim, 2017) . In particular, early question answering systems used expert-crafted ontologies in order to endow these systems with common knowledge (Harabagiu et al., 2005; Xu et al., 2016). …

Content truncated to 500 words