Beyond Context: A New Perspective for Word Embeddings

Proceedings of the Eighth Joint Conference on Lexical and Computational Semantics (*SEM), pages 22–32 Minneapolis, June 6–7, 2019. c⃝2019 Association for Computational Linguistics 22 Beyond Context: A New Perspective for Word Embeddings Yichu Zhou School of Computing University of Utah flyaway@cs.utah.edu Vivek Srikumar School of Computing University of Utah svivek@cs.utah.edu Abstract Most word embeddings today are trained by optimizing a language modeling goal of scor- ing words in their context, modeled as a multi- class classification problem. Despite the suc- cesses of this assumption, it is incomplete: in addition to its context, orthographical or mor- phological aspects of words can offer clues about their meaning. In this paper, we de- fine a new modeling framework for training word embeddings that captures this intuition. Our framework is based on the well-studied problem of multi-label classification and, con- sequently, exposes several design choices for featurizing words and contexts, loss functions for training and score normalization. Indeed, standard models such as CBOW and FAST- TEXT are specific choices along each of these axes. We show via experiments that by com- bining feature engineering with embedding learning, our method can outperform CBOW using only 10% of the training data in both the standard word embedding evaluations and also text classification experiments. 1 Introduction The distributional hypothesis (Firth, 1935; Harris, 1954) has been a cornerstone in NLP. For exam- ple, Firth (1935) writes: . . . the complete meaning of a word is always contextual, and no study of meaning apart from a complete context can be taken seriously. Operationally, in modern NLP, word embeddings capture this idea and are typically trained using neural language models or word collocations (e.g. Bengio et al., 2003; Collobert and Weston, 2008; Mikolov et al., 2013b; Pennington et al., 2014; Pe- ters et al., 2018; Devlin et al., 2018). Is word meaning exclusively defined by its con- text? In this paper, we argue that while the word usage plays a crucial role in defining its meaning (perhaps, centrally so), it is not the only mecha- nism that endows meaning to words. Indeed, Firth writes in the paragraph before the above quote: ...a certain component of the mean- ing of a word is described when you say what sort of a word it is, that is when you identify it morphologically... The composition of a word, (i.e., its orthography and morphology) may offer cues about its mean- ing even if the word is not commonly used, thus allowing us to understand unseen words. For ex- ample, we can elide over misspellings of words (e.g., Bbeijing) because we observe the similari- ties in the orthography between words. By ignoring word-level information, many ex- isting off-the-shelf word embedding approaches suffer from two shortcomings. First, they need a great amount of training data to get high qual- ity word embeddings. Second, even with large amounts of training data, some words (e.g., neol- ogisms, misspellings, technical terms) will not be seen frequently enough to provide statistical sup- port for good embeddings. In this paper, …

Content truncated to 500 words