Orthogonality of Syntax and Semantics within Distributional Spaces

Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing, pages 1301–1310, Beijing, China, July 26-31, 2015. c⃝2015 Association for Computational Linguistics Orthogonality of Syntax and Semantics within Distributional Spaces Jeff Mitchell School of Informatics University of Edinburgh Edinburgh, EH8 9AB, UK jeff.mitchell@ed.ac.uk Mark Steedman School of Informatics University of Edinburgh Edinburgh, EH8 9AB, UK steedman@inf.ed.ac.uk Abstract A recent distributional approach to word- analogy problems (Mikolov et al., 2013b) exploits interesting regularities in the structure of the space of representations. Investigating further, we find that per- formance on this task can be related to orthogonality within the space. Explic- itly designing such structure into a neu- ral network model results in represen- tations that decompose into orthogonal semantic and syntactic subspaces. We demonstrate that learning from word-order and morphological structure within En- glish Wikipedia text to enable this de- composition can produce substantial im- provements on semantic-similarity, pos- induction and word-analogy tasks. 1 Introduction Distributional methods have become widely used across computational linguistics. Recent applica- tions include predicate clustering for question an- swering (Lewis and Steedman, 2013), bilingual embeddings for machine translation (Zou et al., 2013) and enhancing the coverage of POS tag- ging (Huang et al., 2013). The popularity of these methods, stemming from their conceptual simplic- ity and wide applicability, motivates a deeper anal- ysis of the structure of the representations they produce. Commonly, these representations are made in a single vector space with similarity being the main structure of interest. However, recent work by Mikolov et al. (2013b) on a word-analogy task suggests that such spaces may have further use- ful internal regularities. They found that seman- tic differences, such as between big and small, and also syntactic differences, as between big and bigger, were encoded consistently across their space. In particular, they solved the word-analogy problems by exploiting the fact that equivalent re- lations tended to correspond to parallel vector- differences. In this paper, we investigate orthogonality be- tween relations rather than parallelism. While par- allelism serves to ensure that the same relation is encoded consistently, our hypothesis is that or- thogonality serves to ensure that distinct relations are clearly differentiable. We focus specifically on semantic and syntactic relations as these are probably the most distinct classes of properties en- coded in distributional spaces. Empirically, we demonstrate that orthogonal- ity predicts performance on the word-analogy task for three existing approaches to constructing word vectors. We also attempt to enhance the weak- est of these three models by imposing an orthog- onal structure in its construction. In these exten- sions, word representations decompose into or- thogonal semantic and syntactic spaces, and we use word-order and morphology to drive this sep- aration. This decomposition also allows us to de- fine a novel approach to solving the word-analogy problems and our extended models become com- petitive with the other two original models. In addition, we show that the separate semantic and syntactic sub-spaces gain improved performance on …

Content truncated to 500 words