Bayesian Hierarchical Words Representation Learning

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3871–3877 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 3871 Abstract This paper presents the Bayesian Hierarchical Words Representation (BHWR) learning algorithm. BHWR facilitates Variational Bayes word representation learning combined with semantic taxonomy modeling via hierarchical priors. By propagating relevant information between related words, BHWR utilizes the taxonomy to improve the quality of such representations. Evaluation of several linguistic datasets demonstrates the advantages of BHWR over suitable alternatives that facilitate Bayesian modeling with or without semantic priors. Finally, we further show that BHWR produces better representations for rare words. 1 Introduction In the last decade, a plethora of methods were proposed for learning vector representations for words (Mikolov et al., 2013; Pennington et al., 2014; Barkan, 2017), sentences (Lin et al, 2017; Barkan et al., 2020a), items (Barkan et al., 2016; Barkan et al., 2019; Barkan et al., 2020b; Barkan et al., 2020c), and medical concepts (Luo, el al., 2019). In the domain of natural language understanding, neural word embedding models are designed to learn distributed word representations as vectors in a latent space. In this space, arithmetic operations between the word * Equal contribution. vectors encode semantic and syntactic information. Specifically, the seminal works by (Mikolov et al., 2013; Pennington et al., 2014; Bojanowski et al., 2017), exhibited state-of-the-art performance on various linguistic tasks (Finkelstein et al., 2001; Luong et al., 2013; Rogers et al., 2018). The major focus of most previous models was on optimizing the utilization of co-occurrence relations for learning representations, e.g., learning the probability of word 𝑥 to appear in the vicinity of word 𝑦 . Yet, often, additional side information can be leveraged for learning finer embeddings. In this work, we focus on incorporating word semantic taxonomy, which is particularly useful for learning representations of rare words and for learning word representations from a small-size corpus. To this end, we introduce the Bayesian Hierarchical Words Representation (BHWR) learning algorithm. BHWR presents two complementary properties: Bayesian modeling of word representations aside with hierarchical priors that naturally support semantic taxonomy. BHWR is based on a Variational Bayes (VB) optimization that enables the mapping of words into probability densities in the latent space. A key advantage of BHWR is the utilization of word taxonomy for the propagation of relevant information between related words. For example, consider the words ‘anode’ and ‘cathode’. Both words have a common relationship to the word Bayesian Hierarchical Words Representation Learning Oren Barkan1*, Idan Rejwan12*, Avi Caciularu13* and Noam Koenigstein14 1Microsoft, 2School of Computer Science, Tel Aviv University, 3Computer Science Department, Bar-Ilan University, 4Department of Industrial Engineering, Tel Aviv University orenb@microsoft.com, {idanrejwan91, avi.c33}@gmail.com, noamk@tauex.tau.ac.il 3872 ‘electrode’ which appears hierarchically above them in the taxonomy knowledge base. Assume the word ‘cathode’ frequently appears in the corpus, while the words ‘anode’ and ‘electrode’ do not appear in the corpus or occur very infrequently. A model that relies solely on co- occurrence relations will fail to infer the semantic proximity between …

Content truncated to 500 words