Word Rotator’s Distance

Word Rotator’s Distance Sho Yokoi 1,2 Ryo Takahashi 1,2 Reina Akama 1,2 Jun Suzuki 1,2 Kentaro Inui 1,2 1 Tohoku University 2 RIKEN {yokoi, ryo.t, reina.a, jun.suzuki, inui}@ecei.tohoku.ac.jp Abstract A key principle in assessing textual similar- ity is measuring the degree of semantic over- lap between two texts by considering the word alignment. Such alignment-based approaches are intuitive and interpretable; however, they are empirically inferior to the simple cosine similarity between general-purpose sentence vectors. To address this issue, we focus on and demonstrate the fact that the norm of word vectors is a good proxy for word importance, and their angle is a good proxy for word sim- ilarity. Alignment-based approaches do not distinguish them, whereas sentence-vector ap- proaches automatically use the norm as the word importance. Accordingly, we propose a method that first decouples word vectors into their norm and direction, and then com- putes alignment-based similarity using earth mover’s distance (i.e., optimal transport cost), which we refer to as word rotator’s distance. Besides, we find how to “grow” the norm and direction of word vectors (vector con- verter), which is a new systematic approach derived from sentence-vector estimation meth- ods. On several textual similarity datasets, the combination of these simple proposed meth- ods outperformed not only alignment-based approaches but also strong baselines. 1 1 Introduction This paper addresses the semantic textual similar- ity (STS) task, the goal of which is to measure the degree of semantic equivalence between two sentences (Agirre et al., 2012). High-quality STS methods can be used to upgrade the loss functions and automatic evaluation metrics of text generation tasks because a requirement of these metrics is pre- cisely the calculation of STS (Wieting et al., 2019; Zhao et al., 2019; Zhang et al., 2019). 1The source code is available at https://github.com/ eumesy/wrd There are two major approaches to tackling STS. One is to measure the degree of semantic over- lap between texts by considering the word align- ment, which we refer to as alignment-based ap- proaches (Sultan et al., 2014; Kusner et al., 2015; Zhao et al., 2019). The other approach involves generating general-purpose sentence vectors from two texts (typically comprising word vectors), and then calculating their similarity, which we refer to as sentence-vector approaches (Arora et al., 2017; Ethayarajh, 2018). Alignment-based approaches are consistent with human intuition about textual similarity, and their predictions are interpretable. However, the performance of such approaches is lower than that of sentence-vector approaches. We hypothesize that one reason for the inferior- ity of alignment-based approaches is that they do not separate the norm and direction of the word vectors. In contrast, sentence-vector approaches automatically exploit the norm of the word vectors as the relative importance of words. Thus, we propose an STS method that first de- couples word vectors into their norms and direction vectors and then aligns the direction vectors using earth mover’s distance (EMD). Here, the key idea is to map the norm and angle of the word vectors to the EMD parameters probability mass and …

Content truncated to 500 words