GWU NLP at SemEval-2016 Shared Task 1: Matrix Factorization for Crosslingual STS

Proceedings of SemEval-2016, pages 663–667, San Diego, California, June 16-17, 2016. c⃝2016 Association for Computational Linguistics GWU NLP at SemEval-2016 Shared Task 1: Matrix Factorization for Crosslingual STS Hanan Aldarmaki and Mona Diab Department of Computer Science The George Washington University {aldarmaki;mtdiab}@gwu.edu Abstract We present a matrix factorization model for learning cross-lingual representations for sentences. Using sentence-aligned corpora, the proposed model learns distributed repre- sentations by factoring the given data into language-dependent factors and one shared factor. As a result, input sentences from both languages can be mapped into fixed-length vectors and then compared directly using the cosine similarity measure, which achieves 0.8 Pearson correlation on Spanish-English se- mantic textual similarity. 1 Introduction Semantic textual similarity (STS) is a measure of relatedness in meaning between a pair of variable- length textual snippets, such as sentences. Using unsupervised vector space models, words and sen- tences can be mapped into dense vector represen- tations that capture implicit syntactic and semantic information. These representations can then be di- rectly compared using well-known distance or sim- ilairty measures, such as the Euclidean distance or cosine similarity, which reflect their overall seman- tic relatedness. Such distributed representations of words, or word embeddings, can be learned using global word co-occurrence statistics as in matrix factorization models (Guo and Diab, 2012; Pennington et al., 2014), or using local context as in neural proba- bilistic language models (Bengio et al., 2003; Col- lobert and Weston, 2008; Socher et al., 2013). A variable-length sentence can be mapped into a fixed- length vector either by combining word embeddings or directly learning sentence representations as in the paragraph vector model proposed in (Le and Mikolov, 2014). In crosslingual STS, the challenge is to compare sentences from two different languages. We address this problem by directly learning crosslingual vector representations for words and sentences, which al- lows us to calculate the STS scores without the need for explicit translation or mapping. Several mod- els can be used for learning cross-lingual composi- tional representations (Klementieva et al., 2012; Shi et al., 2015; Pennington et al., 2014; Cavallanti et al., 2010; Mikolov et al., 2013; Coulmance et al., 2015; Pham et al., 2015). We propose a relatively sim- ple and nuanced unsupervised model inspired by the monolingual weighted matrix factorization (WMF) model proposed in (Guo and Diab, 2012), which we extend to the cross-lingual setting. The WMF model learns word representations by decomposing a sparse tf-idf matrix into two low- rank factor matrices representing words and sen- tences. The weights are adjusted to reflect the con- fidence levels in reconstructing observed vs. miss- ing words in the original matrix. Representations for variable-length sequences can be calculated by minimizing the reconstruction error as described in Section 2.1. In this paper, we propose to extend this model to the cross-lingual setting by modeling two languages in parallel to obtain shared semantic representations. The proposed model has a simple loss function and only uses sentence-aligned data for learning the shared representations. We describe the model in two …

Content truncated to 500 words