Morphological Word-Embeddings

Human Language Technologies: The 2015 Annual Conference of the North American Chapter of the ACL, pages 1287–1292, Denver, Colorado, May 31 – June 5, 2015. c⃝2015 Association for Computational Linguistics Morphological Word-Embeddings Ryan Cotterell1,2 Department of Computer Science1 Johns Hopkins University, USA ryan.cotterell@jhu.edu Hinrich Schütze2 Center for Information and Language Processing2 University of Munich, Germany inquiries@cislmu.org Abstract Linguistic similarity is multi-faceted. For in- stance, two words may be similar with re- spect to semantics, syntax, or morphology in- ter alia. Continuous word-embeddings have been shown to capture most of these shades of similarity to some degree. This work con- siders guiding word-embeddings with mor- phologically annotated data, a form of semi- supervised learning, encouraging the vectors to encode a word’s morphology, i.e., words close in the embedded space share morpho- logical features. We extend the log-bilinear model to this end and show that indeed our learned embeddings achieve this, using Ger- man as a case study. 1 Introduction Word representation is fundamental for NLP. Re- cently, continuous word-embeddings have gained traction as a general-purpose representation frame- work. While such embeddings have proven them- selves useful, they typically treat words holistically, ignoring their internal structure. For morphologi- cally impoverished languages, i.e., languages with a low morpheme-per-word ratio such as English, this is often not a problem. However, for the processing of morphologically-rich languages exploiting word- internal structure is necessary. Word-embeddings are typically trained to pro- duce representations that capture linguistic similar- ity. The general idea is that words that are close in the embedding space should be close in meaning. A key issue, however, is that meaning is a multi- faceted concept and thus there are multiple axes, along which two words can be similar. For example, ice and cold are topically related, ice and fire are syntactically related as they are both nouns, and ice and icy are morphologically related as they are both derived from the same root. In this work, we are interested in distinguishing between these various axes and guiding the embeddings such that similar embeddings are morphologically related. We augment the log-bilinear model (LBL) of Mnih and Hinton (2007) with a multi-task objective. In addition to raw text, our model is trained on a corpus annotated with morphological tags, encour- aging the vectors to encode a word’s morphology. To be concrete, the first task is language modeling— the traditional use of the LBL—and the second is akin to unigram morphological tagging. The LBL, described in section 3, is fundamentally a language model (LM)—word-embeddings fall out as low di- mensional representations of context used to pre- dict the next word. We extend the model to jointly predict the next morphological tag along with the next word, encouraging the resulting embeddings to encode morphology. We present a novel met- ric and experiments on German as a case study that demonstrates that our approach produces word- embeddings that better preserve morphological rela- tionships. 2 Related Work Here we discuss the role morphology has played in language modeling and offer …

Content truncated to 500 words