CNRC at SemEval-2016 Task 1: Experiments in Crosslingual Semantic Textual Similarity

Proceedings of SemEval-2016, pages 668–673, San Diego, California, June 16-17, 2016. c⃝2016 Association for Computational Linguistics CNRC at SemEval-2016 Task 1: Experiments in Crosslingual Semantic Textual Similarity Chi-kiu Lo Cyril Goutte National Research Council Canada Multilingual Text Processing 1200 Montreal Road, Ottawa, Ontario K1A 0R6, Canada FirstName.LastName@nrc.ca Michel Simard Abstract We describe the systems entered by the National Research Council Canada in the SemEval-2016 Task1: Crosslingual Semantic Textual Similarity. We tried two approaches: One computes a true crosslingual similarity based on features extracted from lexical se- mantics and shallow semantic structures of the source and target fragments, combined using a linear model. The other approach relies on Statistical Machine Translation, followed by a monolingual semantic similarity, relying again on syntactic and semantic features. We report our experiments using trial data, as well as official final results on the evaluation data. 1 Introduction The SemEval-2016 Semantic Textual Similarity (STS) evaluation (task1) introduced a crosslingual track. Given a Spanish-English bilingual fragment pair, the goal is to compute the degree of equiva- lence between them. This offers additional chal- lenges compared to the “STS Core” track, where both fragments are in the same language (English or Spanish in 2014-15). The crosslingual track requires potentially to detect which fragment is in which lan- guage, perform further language processing accord- ingly, and estimate lexical and semantic similarities across languages. In our work, we investigated two approaches. In the first approach, we try to build a true crosslingual similarity based on a number of features computed from both fragments. One of these features projects Spanish words into an English embedding space in order to compute similarities in that space. Other features compute various kinds of syntactic and se- mantic overlap between the fragments. These fea- tures are combined in a linear model estimated on the trial data, and combined with an isotonic regres- sion (de Leeuw et al., 2009) layer in order to account for non-linearity in the scores. The second approach uses a Statistical Machine Translation system to map Spanish fragments to En- glish, then relies on a monolingual semantic similar- ity between the translated fragment and the English fragment. Various monolingual similarity features, using embeddings, syntactic and semantic informa- tion, are computed and combined again using a lin- ear model followed by an isotonic regression layer. In the next section, we describe the two ap- proaches and their components: SMT system, crosslingual and monolingual feature extraction, and the output layer fitting the features to the seman- tic similarity scores. We then describe the corpora used to fit the features to the output similarity score, and make various modeling choices (Section 3). We present our experimental results on the trial and test data in Section 4. 2 System description We describe our two approaches: the direct crosslin- gual similarity using embedding mapping (EMAP run) and the use of Machine Translation followed by a monolingual semantic similarity (MT1 and MT2). 2.1 Crosslingual Embedding Mapping Feature Extraction: We evaluate the semantic similarity of the given text based …

Content truncated to 500 words