FBK HLT-MT at SemEval-2016 Task 1: Cross-lingual Semantic Similarity Measurement Using Quality Estimation Features and Compositional Bilingual Word Embeddings

Proceedings of SemEval-2016, pages 570–576, San Diego, California, June 16-17, 2016. c⃝2016 Association for Computational Linguistics FBK HLT-MT at SemEval-2016 Task 1: Cross-lingual Semantic Similarity Measurement Using Quality Estimation Features and Compositional Bilingual Word Embeddings Duygu Ataman1,2, Jos´e G. C. de Souza1,2, Marco Turchi1, Matteo Negri1 1 HLT - MT Fondazione Bruno Kessler, Trento, Italy 2Department of Information Engineering and Computer Science University of Trento, Italy {ataman,desouza,turchi,negri}@fbk.eu Abstract This paper describes the system by FBK HLT- MT for cross-lingual semantic textual similar- ity measurement. Our approach is based on supervised regression with an ensemble deci- sion tree. In order to assign a semantic similar- ity score to an input sentence pair, the model combines features collected by state-of-the-art methods in machine translation quality esti- mation and distance metrics between cross- lingual embeddings of the two sentences. In our analysis, we compare different techniques for composing sentence vectors, several dis- tance features and ways to produce training data. The proposed system achieves a mean Pearson’s correlation of 0.39533, ranking 7th among all participants in the cross-lingual STS task organized within the SemEval 2016 evaluation campaign. 1 Introduction Semantic textual similarity (STS) measures the de- gree of equivalence between the meanings of two text sequences (Agirre et al., 2015). The similarity of the text pair can be represented as a continuous or discrete-time value ranging from irrelevance to exact semantic equivalence (Agirre et al., 2015). STS has been one of the official shared tasks in SemEval since 2013 and has attracted the par- ticipation of many researchers from the scientific community; enabling the evaluation of several dif- ferent approaches in natural language processing with a common benchmark and the production of novel annotated data sets that can be used in future research. State-of-the-art monolingual STS meth- ods make use of several approaches including word alignments and distributional semantics, which are typically employed in a machine learning scenario (Sultan et al., 2015; H¨anig et al., 2015). This is the first year in which SemEval has or- ganized a cross-lingual STS (CL-STS) sub-task, for which a baseline system applicable to the problem has not been defined yet. Similar to the monolin- gual STS task, the cross-lingual task requires the in- terpretation of the semantic similarity of two cross- lingual sentences, one in English and another one in Spanish, with a score ranging from 0 to 5. CL- STS measurement could be extremely useful for achieving textual entailment, paraphrase identifica- tion, word-sense disambiguation or sentiment anal- ysis at the cross-lingual level as well as providing new means for an adequacy-oriented evaluation of machine translation outputs. A related task in natural language processing is quality estimation. Quality estimation (QE) is used for automatically predicting the quality of machine translation outputs with respect to the source sen- tences in the original language (Mehdad et al., 2012; Turchi et al., 2014; C. de Souza et al., 2014a; C. de Souza et al., 2014b; C. de Souza et al., 2015). One shortcoming of QE approaches is that the QE …

Content truncated to 500 words