Learning Word Representations by Jointly Modeling Syntagmatic and Paradigmatic Relations

Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing, pages 136–145, Beijing, China, July 26-31, 2015. c⃝2015 Association for Computational Linguistics Learning Word Representations by Jointly Modeling Syntagmatic and Paradigmatic Relations Fei Sun, Jiafeng Guo, Yanyan Lan, Jun Xu, and Xueqi Cheng CAS Key Lab of Network Data Science and Technology Institute of Computing Technology Chinese Academy of Sciences, China ofey.sunfei@gmail.com {guojiafeng,lanyanyan,junxu,cxq}@ict.ac.cn Abstract Vector space representation of words has been widely used to capture fine-grained linguistic regularities, and proven to be successful in various natural language pro- cessing tasks in recent years. However, existing models for learning word repre- sentations focus on either syntagmatic or paradigmatic relations alone. In this pa- per, we argue that it is beneficial to jointly modeling both relations so that we can not only encode different types of linguistic properties in a unified way, but also boost the representation learning due to the mu- tual enhancement between these two types of relations. We propose two novel dis- tributional models for word representation using both syntagmatic and paradigmatic relations via a joint training objective. The proposed models are trained on a public Wikipedia corpus, and the learned rep- resentations are evaluated on word anal- ogy and word similarity tasks. The re- sults demonstrate that the proposed mod- els can perform significantly better than all the state-of-the-art baseline methods on both tasks. 1 Introduction Vector space models of language represent each word with a real-valued vector that captures both semantic and syntactic information of the word. The representations can be used as basic features in a variety of applications, such as information re- trieval (Manning et al., 2008), named entity recog- nition (Collobert et al., 2011), question answer- ing (Tellex et al., 2003), disambiguation (Sch¨utze, 1998), and parsing (Socher et al., 2011). A common paradigm for acquiring such repre- sentations is based on the distributional hypothe- sis (Harris, 1954; Firth, 1957), which states that is wolf The a fierce animal. is tiger The a fierce animal. syntagmatic syntagmatic paradigmatic Figure 1: Example for syntagmatic and paradig- matic relations. words occurring in similar contexts tend to have similar meanings. Based on this hypothesis, vari- ous models on learning word representations have been proposed during the last two decades. According to the leveraged distributional infor- mation, existing models can be grouped into two categories (Sahlgren, 2008). The first category mainly concerns the syntagmatic relations among the words, which relate the words that co-occur in the same text region. For example, “wolf” is close to “fierce” since they often co-occur in a sen- tence, as shown in Figure 1. This type of models learn the distributional representations of words based on the text region that the words occur in, as exemplified by Latent Semantic Analysis (LSA) model (Deerwester et al., 1990) and Non-negative Matrix Factorization (NMF) model (Lee and Se- ung, 1999). The second category mainly cap- tures paradigmatic relations, which relate words that occur with similar …

Content truncated to 500 words