Investigating Language Universal and Specific Properties in Word Embeddings

Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pages 1478–1488, Berlin, Germany, August 7-12, 2016. c⃝2016 Association for Computational Linguistics Investigating Language Universal and Specific Properties in Word Embeddings Peng Qian Xipeng Qiu∗ Xuanjing Huang Shanghai Key Laboratory of Intelligent Information Processing, Fudan University School of Computer Science, Fudan University 825 Zhangheng Road, Shanghai, China {pqian11, xpqiu, xjhuang}@fudan.edu.cn Abstract Recently, many NLP tasks have benefit- ed from distributed word representation. However, it remains unknown whether embedding models are really immune to the typological diversity of languages, despite the language-independent archi- tecture. Here we investigate three repre- sentative models on a large set of language samples by mapping dense embedding to sparse linguistic property space. Experi- ment results reveal the language universal and specific properties encoded in various word representation. Additionally, strong evidence supports the utility of word form, especially for inflectional languages. 1 Introduction Word representation is a core issue in natural language processing. Context-based word rep- resentation, which is inspired by Harris (1954), has achieved huge successes in many NLP ap- plications. Despite its popularity, character-based approach also comes out as an equal competitor (Santos and Zadrozny, 2014; Kim et al., 2016; Ling et al., 2015b; Ling et al., 2015a; Faruqui et al., 2016; Ballesteros et al., 2015) . Moreover, questions arise when we consider what these models could capture from linguistic cues under the perspective of cross-language typological di- versity, as is argued by Bender (2009). Despite previous efforts in empirically inter- preting word embedding and exploring the in- trinsic/extrinsic factors in learning process (An- dreas and Klein, 2014; Lai et al., 2015; K¨ohn, 2015; Melamud et al., 2016), it remains unknown whether embedding models are really immune to the structural variance of languages. ∗Corresponding author. Current research has gaps for understanding model behaviours towards language typological diversity as well as the utility of context and form for different languages. Thus, we select three representative types of models and design a series of experiments to reveal the universals and specifics of various word representations on decoding linguistic properties. Our work contributes to shedding new insights into the following topics: a) How do typological differences of language structure influence a word embedding mod- el? Does a model behave similarly towards phylogenetically-related languages? b) Is word form a more efficient predictor of a cer- tain grammatical function than word context for specific languages? c) How do the neurons of a model respond to linguistic features? Can we explain the utility of context and form by analyzing neuron activation pattern? 2 Experiment Design To study the proposed questions above, we design four series of experiments to comprehensively compare context-based and character-based word representations on different languages, covering syntactic, morphological and semantic properties. The basic paradigm is to decode interpretable linguistic features from a target collection of word representations. We hypothesize that there exists a linear/nonlinear map between a word representation x and a high-level sparse feature vector y if the word vector implicitly encode sufficient information1. Figure1 …

Content truncated to 500 words