Word2Box: Capturing Set-Theoretic Semantics of Words using Box Embeddings

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 2263 - 2276 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Word2Box: Capturing Set-Theoretic Semantics of Words using Box Embeddings Shib Sankar Dasgupta∗, Michael Boratko∗, Siddhartha Mishra , Shriya Atmakuri , Dhruvesh Patel , Xiang Lorraine Li , Andrew McCallum Manning College of Information & Computer Sciences University of Massachusetts Amherst {ssdasgupta,mboratkro,siddharthami,satmakuri, dhruveshpate,xiangl,mccallum}@cs.umass.edu Abstract Learning representations of words in a con- tinuous space is perhaps the most funda- mental task in NLP, however words interact in ways much richer than vector dot prod- uct similarity can provide. Many relation- ships between words can be expressed set- theoretically, for example adjective-noun com- pounds (eg. “red cars”⊆“cars”) and homo- graphs (eg. “tongue”∩“body” should be sim- ilar to “mouth”, while “tongue”∩“language” should be similar to “dialect”) have natural set-theoretic interpretations. Box embeddings are a novel region-based representation which provide the capability to perform these set- theoretic operations. In this work, we provide a fuzzy-set interpretation of box embeddings, and learn box representations of words using a set-theoretic training objective. We demon- strate improved performance on various word similarity tasks, particularly on less common words, and perform a quantitative and qualita- tive analysis exploring the additional unique expressivity provided by WORD2BOX. 1 Introduction The concept of learning a distributed representa- tion for a word has fundamentally changed the field of natural language processing. The introduction of efficient methods for training vector representa- tions of words in Word2Vec (Mikolov et al., 2013), and later GloVe (Pennington et al.) as well as Fast- Text (Bojanowski et al., 2017) revolutionized the field, paving the way for the recent wave of deep architectures for language modeling, all of which implicitly rely on this fundamental notion that a word can be effectively represented by a vector. While now ubiquitous, the concept of represent- ing a word as a single point in space is not partic- ularly natural. All senses and contexts, levels of abstraction, variants and modifications which the word may represent are forced to be captured by ∗*Equal Contributions. Lorem ipsum dolor sit amet, consectetur adipiscing elit. Curabitur viverra pretium diam, nec sodales urna semper non. Nunc ut semper arcu. Vestibulum non fringilla lacus. Nullam iaculis leo hendrerit arcu finibus imperdiet. Proin eget quam sit amet mi tempus interdum ac at sapien. Nulla facilisi. Aenean pretium orci a augue maximus malesuada. Quisque pretium, dui non tempor luctus, magna massa varius sapien, consectetur dictum velit dolor elementum mi. Phasellus vel dui non est accumsan gravida. Morbi blandit diam eget orci posuere, in euismod purus finibus. Vestibulum condimentum id ipsum dictum porta. Aenean in lectus nisi. Pellentesque elementum elit ac risus pretium vehicula. Curabitur ut dapibus mi. Fusce in volutpat felis. Morbi at euismod elit. Vestibulum malesuada quam quis justo porta consectetur. Duis mollis libero eget sapien aliquet, at vehicula tellus consectetur. Maecenas id auctor nisi, id aliquam tellus. Quisque lectus elit, viverra at elementum a, imperdiet a lorem. Vestibulum in justo sit …

Content truncated to 500 words