Cross-Topic Distributional Semantic Representations Via Unsupervised Mappings

Proceedings of NAACL-HLT 2019, pages 1052–1061 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 1052 Cross-Topic Distributional Semantic Representations Via Unsupervised Mappings Eleftheria Briakou1,2∗, Nikos Athanasiou2, Alexandros Potamianos2,3 1University of Maryland, College Park, MD 2School of ECE, National Technical University of Athens, Athens, Greece 3Signal Analysis and Interpretation Laboratory (SAIL), USC, Los Angeles, USA ebriakou@cs.umd.edu, athn.nik@gmail.com, potam@central.ntua.gr Abstract In traditional Distributional Semantic Mod- els (DSMs) the multiple senses of a polyse- mous word are conflated into a single vector space representation. In this work, we pro- pose a DSM that learns multiple distributional representations of a word based on different topics. First, a separate DSM is trained for each topic and then each of the topic-based DSMs is aligned to a common vector space. Our unsupervised mapping approach is moti- vated by the hypothesis that words preserv- ing their relative distances in different topic semantic sub-spaces constitute robust seman- tic anchors that define the mappings between them. Aligned cross-topic representations achieve state-of-the-art results for the task of contextual word similarity. Furthermore, eval- uation on NLP downstream tasks shows that multiple topic-based embeddings outperform single-prototype models. 1 Introduction Word-level representation learning algorithms adopt the distributional hypothesis (Harris, 1954), presuming a correlation between the distributional and the semantic relationships of words. Typi- cally, these models encode the contextual infor- mation of words into dense feature vectors—often referred to as embeddings—of a k-dimensional space, thus creating a Vector Space Model (VSM) of lexical semantics. Such embeddings have been successfully applied to various natural language processing applications, including information re- trieval (Manning et al., 2008), sentiment analysis (Tai et al., 2015), and machine translation (Amiri et al., 2016; Sharaf et al., 2017). Despite their popularity, traditional DSMs rely solely on models where each word is uniquely rep- resented by one point in the vector space. From a ∗The research was performed when the author was an un- dergraduate researcher at School of ECE, NTUA in Athens, Greece. linguistic perspective, these models cannot capture the distinct meanings of polysemous words (e.g., bank or cancer), resulting in conflated word rep- resentations of diverse contextual semantics. To alleviate this problem, DSMs with multi- ple representations per word have been proposed in the literature, based on clustering local con- texts of individual words (Reisinger and Mooney, 2010; Tian et al., 2014; Neelakantan et al., 2014). An alternative way to train multiple representa- tion DSMs is to utilize semantic lexical resources (Rothe and Sch¨utze, 2015; Pilehvar and Collier, 2016). Christopoulou et al. (2018), based on the assumption that typically words appear with a spe- cific sense in each topic, proposed a topic-based semantic mixture model that exploits a combina- tion of similarities estimated on topic-based DSMs for the computation of semantic similarity be- tween words. Their model performs well for a va- riety of semantic similarity tasks; however, it lacks a unified representation of multiple senses in a common semantic space. The problem of defining transformations between embeddings—trained in- dependently under different corpora—has been …

Content truncated to 500 words