Learning Lexical Subspaces in a Distributional Vector Space

Learning Lexical Subspaces in a Distributional Vector Space Kushal Arora∗ Aishik Chakraborty∗ Jackie C. K. Cheung School of Computer Science, McGill University Qu´ebec AI Instuite (Mila) {kushal.arora,aishik.chakraborty}@mail.mcgill.ca, jcheung@cs.mcgill.ca Abstract In this paper, we propose LEXSUB, a novel approach towards unifying lexical and dis- tributional semantics. We inject knowledge about lexical-semantic relations into distribu- tional word embeddings by defining subspaces of the distributional vector space in which a lexical relation should hold. Our frame- work can handle symmetric attract and repel relations (e.g., synonymy and antonymy, respectively), as well as asymmetric relations (e.g., hypernymy and meronomy). In a suite of intrinsic benchmarks, we show that our model outperforms previous approaches on related- ness tasks and on hypernymy classification and detection, while being competitive on word similarity tasks. It also outperforms previous systems on extrinsic classification tasks that benefit from exploiting lexical relational cues. We perform a series of analyses to understand the behaviors of our model.1 1 Introduction Pre-trained word embeddings are the bedrock of modern natural language processing architectures. This success of pre-trained word embeddings is attributed to their ability to embody the distributional hypothesis (Harris, 1954; Firth, 1957), which states that ‘‘the words that are used in the same contexts tend to purport similar meanings’’ (Harris, 1954). Thebiggeststrengthoftheembedding methods— their ability to cluster distributionally related words—is also their biggest weakness. This contextual clustering of words brings together words that might be used in a similar context in the ∗Equal contribution. 1Code available at https://github.com/ aishikchakraborty/LexSub. text, but that might not necessarily be semantically similar, or worse, might even be antonyms (Lin et al., 2003). Several techniques have been proposed in the literature to modify word vectors to incorporate lexical-semantic relations into the embedding space (Yu and Dredze, 2014; Xu et al., 2014; Fried and Duh, 2014; Faruqui et al., 2015; Mrkˇsi´c et al., 2016; Mrkˇsi´c et al., 2017; Glavaˇs and Vuli´c, 2018). The common theme of these approaches is that they modify the original distributional vector space using auxiliary lexical constraints to endow the vector space with a sense of lexical relations. However, a potential limitation of this approach is that the alteration of the original distributional space may cause a loss of the distributional information that made these vectors so useful in the first place, leading to degraded performance when used in the downstream tasks. This problemcould be further exacerbated when multiple relations are incorporated, especially as different lexical-semantic relations have different mathematical properties. For example, synonymy is a symmetric relation, whereas hypernymy and meronymy are asymmetric relations. It would be difficult to control the interacting effects that constraints induced by multiple relations could have on the distributional space. The solution that we propose is to enforce a separation of concerns, in which distributional information is addressed by a central main vector space, whereas each lexical relation is handled by a separate subspace of the main distributional space. The interface between these components is then a projection operation from the main distributional space into a lexical subspace. Our …

Content truncated to 500 words