Polar Embedding

Proceedings of the 25th Conference on Computational Natural Language Learning (CoNLL), pages 470–480 November 10–11, 2021. ©2021 Association for Computational Linguistics 470 Polar Embedding Ran Iwamoto∗, Ryosuke Kohita†,Akifumi Wachi† ∗ Keio University, † IBM Research - Tokyo raniwamoto@gmail.com, kohi@ibm.com, akifumi.wachi@ibm.com Abstract Hierarchical relationships are invaluable infor- mation for many natural language processing (NLP) tasks. Distributional representation has become a fundamental approach for encoding word relationships, particularly embeddings in hyperbolic space showed great performance in representing hierarchies by taking advan- tage of their spatial properties. However, most machine learning systems do not suppose to use in such complex non-Euclidean geome- tries. To achieve hierarchy representations in commonly used Euclidean space, we propose Polar Embedding that learns word embeddings with the polar coordinate system. Utilizing characteristics of polar coordinates, the hier- archy of words is expressed with two inde- pendent variables: radius (generality) and an- gles (similarity), and their variables are op- timized separately. Polar embedding shows word hierarchies explicitly and allows us to use beneficial resources such as word frequen- cies or word generality annotations for com- puting radiuses. We introduce an optimiza- tion method for learning angles in limited ranges of polar coordinates, which combin- ing a loss function controlling gradient and distribution uniformization. Experimental re- sults on hypernymy datasets indicate that our approach outperforms other embeddings in low-dimensional Euclidean space and compet- itively performs even with hyperbolic embed- dings, which possess a geometric advantage. 1 Introduction A hierarchy is structured information that enables us to understand a specific object in a general sense (e.g., dog is one instance of mammal). Such gen- eralization capability is or will be a basis of in- telligent systems such as comprehending causal- ity (Hassanzadeh et al., 2019), common sense (Tal- mor et al., 2019), and logic (Yang et al., 2017). For example, species information (e.g., carnivora vs. herbivore) will be useful when predicting behavior Figure 1: Conceptual illustration representing struc- tures of word hierarchies. of animals. Another example is when developing a question answering system. Hierarchical rela- tions of words enable the system to cover diverse inputs from a user (e.g., how many paw pads does a (cat | kitten | tabby) → (cat) have?). Therefore, to deploy hierarchical information in such systems, it is critical to represent word generality and meaning efficiently in a machine-readable manner. For encoding word relationships, distributional representations are commonly used in natural lan- guage processing. Word embeddings such as Word2Vec (Mikolov et al., 2013), Glove (Penning- ton et al., 2014), and FastText (Bojanowski et al., 2017) express word similarities as continuous vec- tors in Euclidean space, and have brought signif- icant advances to various applications (Collobert et al., 2011; Lample et al., 2016). Hierarchy-aware representations in Euclidean space have also been developed. Gaussian embedding (Vilnis and McCallum, 2015) represents words as Gaussian distributions, whose mean vectors (50–100 dimension) encode word similarity, and whose variances encode word generality. Mean vectors have a similar role to existing word vectors, and word generalities are expressed by adding variance. On the …

Content truncated to 500 words