Exploiting Morphological Regularities in Distributional Word Representations

Exploiting Morphological Regularities in Distributional Word Representations Syed Sarfaraz Akhtar* Arihant Gupta* Avijit Vajpayee* Arjit Srivastava Madan Gopal Jhanwar Manish Shrivastava {syed.akhtar, arihant.gupta, arjit.srivastava, madangopal.jhanwar}@research.iiit.ac.in, avijit@inshorts.com, manish.shrivastava@iiit.ac.in Language Technologies Research Center(LTRC) Kohli Center On Intelligent Systems (KCIS) International Institute of Information Technology Hyderabad Abstract We present a simple, fast and unsuper- vised approach for exploiting morpho- logical regularities present in high di- mensional vector spaces. We propose a novel method for generating embeddings of words from their morphological vari- ants using morphological transformation operators. We evaluate this approach on MSR word analogy test set (Mikolov et al., 2013d) with an accuracy of 85% which is 12% higher than the previous best known system. 1 Introduction Vector representation of words are presently be- ing used to solve a variety of problems like doc- ument classification (Sebastiani, 2002), question answering (Tellex et al., 2003) and chunking (Turian et al., 2010). Word representations capture both syntactic and semantic properties (Mikolov et al., 2013d) of nat- ural language. Soricut and Och (2015) exploited these regularities to generate prefix/suffix based morphological transformation rules in an unsuper- vised manner. These morphological transforma- tions were represented as vectors in the same em- bedding space as the vocabulary. They used a graph based approach and represented transforma- tions as “type:from:to” triples and a direction vec- tor: for example “suffix:ion:e:↑creation” implies a suffix change just like in the case “creation” to “create”. Using Soricut’s transformation rules, the major problem is identifying the correct direction vector to use for a given case, i.e. if we have to generate an embedding for “runs”, which rule to apply on “run”. Experimental results showed that “walk - * These authors contributed equally to this work. walks” gives better results than rules like “invent - invents” or “object - objects” in generating word embedding for “runs”. In this paper, we try to ex- plore if we can harness this morphological regu- larity in a much better way, than applying a single direction using vector arithmetic. Hence, we tried to come up with a global trans- formation operator, which aligns itself with the source word, to give best possible word embed- ding for target word. We will have a single trans- formation operator for each rule, irrespective of the form of root word (like verb or a noun). Our transformation operator is in the form of a ma- trix, which when applied on a word embedding (cross product of vector representation of word with transformation matrix) gives us a word em- bedding for target word. The intuition is not to solve for “invent is to in- vents as run is to ?” or ”object is to objects as run is to ?”, but instead we are solving for “walk is to walks, object is to objects, invent is to invents, .... as run is to ?”. A transformation operator aims to be a unified transition function for different forms of the same transition. Learning a representation of this operator would allow us to capture the …

Content truncated to 500 words