Semeval-2013 Task 8: Cross-lingual Textual Entailment for Content Synchronization

Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Seventh International Workshop on Semantic Evaluation (SemEval 2013), pages 25–33, Atlanta, Georgia, June 14-15, 2013. c⃝2013 Association for Computational Linguistics Semeval-2013 Task 8: Cross-lingual Textual Entailment for Content Synchronization Matteo Negri FBK-irst Trento, Italy negri@fbk.eu Alessandro Marchetti CELCT Trento, Italy amarchetti@celct.it Yashar Mehdad UBC Vancouver, Canada mehdad@cs.ubc.ca Luisa Bentivogli FBK-irst Trento, Italy bentivo@fbk.eu Danilo Giampiccolo CELCT Trento, Italy giampiccolo@celct.it Abstract This paper presents the second round of the task on Cross-lingual Textual Entailment for Content Synchronization, organized within SemEval-2013. The task was designed to pro- mote research on semantic inference over texts written in different languages, targeting at the same time a real application scenario. Par- ticipants were presented with datasets for dif- ferent language pairs, where multi-directional entailment relations (“forward”, “backward”, “bidirectional”, “no entailment”) had to be identified. We report on the training and test data used for evaluation, the process of their creation, the participating systems (six teams, 61 runs), the approaches adopted and the re- sults achieved. 1 Introduction The cross-lingual textual entailment task (Mehdad et al., 2010) addresses textual entailment (TE) recog- nition (Dagan and Glickman, 2004) under the new dimension of cross-linguality, and within the new challenging application scenario of content synchro- nization. Given two texts in different languages, the cross-lingual textual entailment (CLTE) task con- sists of deciding if the meaning of one text can be inferred from the meaning of the other text. Cross- linguality represents an interesting direction for re- search on recognizing textual entailment (RTE), es- pecially due to its possible application in a vari- ety of tasks. Among others (e.g. question answer- ing, information retrieval, information extraction, and document summarization), multilingual content synchronization represents a challenging application scenario to evaluate CLTE recognition components geared to the identification of sentence-level seman- tic relations. Given two documents about the same topic writ- ten in different languages (e.g. Wikipedia pages), the content synchronization task consists of au- tomatically detecting and resolving differences in the information they provide, in order to produce aligned, mutually enriched versions of the two docu- ments (Monz et al., 2011; Bronner et al., 2012). To- wards this objective, a crucial requirement is to iden- tify the information in one page that is either equiv- alent or novel (more informative) with respect to the content of the other. The task can be naturally cast as an entailment recognition problem, where bidi- rectional and unidirectional entailment judgements for two text fragments are respectively mapped into judgements about semantic equivalence and novelty. The task can also be seen as a machine translation evaluation problem, where judgements about se- mantic equivalence and novelty depend on the pos- sibility to fully or partially translate a text fragment into the other. The recent advances on monolingual TE on the one hand, and the methodologies used in Statisti- cal Machine Translation (SMT) on the other, offer promising solutions to approach the CLTE task. In line with a number of systems that model the RTE task …

Content truncated to 500 words