CompiLIG at SemEval-2017 Task 1: Cross-Language Plagiarism Detection Methods for Semantic Textual Similarity

Proceedings of the 11th International Workshop on Semantic Evaluations (SemEval-2017), pages 109–114, Vancouver, Canada, August 3 - 4, 2017. c⃝2017 Association for Computational Linguistics CompiLIG at SemEval-2017 Task 1: Cross-Language Plagiarism Detection Methods for Semantic Textual Similarity J´er´emy Ferrero Compilatio 276 rue du Mont Blanc 74540 Saint-F´elix, France LIG-GETALP Univ. Grenoble Alpes, France jeremy.ferrero@imag.fr Laurent Besacier LIG-GETALP Univ. Grenoble Alpes, France laurent.besacier@imag.fr Didier Schwab LIG-GETALP Univ. Grenoble Alpes, France didier.schwab@imag.fr Fr´ed´eric Agn`es Compilatio 276 rue du Mont Blanc 74540 Saint-F´elix, France frederic@compilatio.net Abstract We present our submitted systems for Se- mantic Textual Similarity (STS) Track 4 at SemEval-2017. Given a pair of Spanish- English sentences, each system must esti- mate their semantic similarity by a score between 0 and 5. In our submission, we use syntax-based, dictionary-based, context-based, and MT-based methods. We also combine these methods in unsu- pervised and supervised way. Our best run ranked 1st on track 4a with a correlation of 83.02% with human annotations. 1 Introduction CompiLIG is a collaboration between Compilatio1 - a company particularly interested in cross- language plagiarism detection - and LIG research group on natural language processing (GETALP). Cross-language semantic textual similarity detec- tion is an important step for cross-language plagia- rism detection, and evaluation campaigns in this new domain are rare. For the first time, SemEval STS task (Agirre et al., 2016) was extended with a Spanish-English cross-lingual sub-task in 2016. This year, sub-task was renewed under track 4 (di- vided in two sub-corpora: track 4a and track 4b). Given a sentence in Spanish and a sentence in English, the objective is to compute their seman- tic textual similarity according to a score from 0 1www.compilatio.net to 5, where 0 means no similarity and 5 means full semantic similarity. The evaluation metric is a Pearson correlation coefficient between the sub- mitted scores and the gold standard scores from human annotators. Last year, among 26 submis- sions from 10 teams, the method that achieved the best performance (Brychcin and Svoboda, 2016) was a supervised system (SVM regression with RBF kernel) based on word alignment algorithm presented in Sultan et al. (2015). Our submission in 2017 is based on cross- language plagiarism detection methods combined with the best performing STS detection method published in 2016. CompiLIG team participated to SemEval STS for the first time in 2017. The meth- ods proposed are syntax-based, dictionary-based, context-based, and MT-based. They show addi- tive value when combined. The submitted runs consist in (1) our best single unsupervised ap- proach (2) an unsupervised combination of best approaches (3) a fine-tuned combination of best approaches. The best of our three runs ranked 1st with a correlation of 83.02% with human annota- tions on track 4a among all submitted systems (51 submissions from 20 teams for this track). Cor- relation results of all participants (including ours) on track 4b were much lower and we try to explain why (and question the validity of track 4b) in the last part of this paper. 109 2 Cross-Language Textual Similarity Detection Methods …

Content truncated to 500 words