Semeval-2012 Task 8: Cross-lingual Textual Entailment for Content Synchronization

First Joint Conference on Lexical and Computational Semantics (*SEM), pages 399–407, Montr´eal, Canada, June 7-8, 2012. c⃝2012 Association for Computational Linguistics Semeval-2012 Task 8: Cross-lingual Textual Entailment for Content Synchronization Matteo Negri FBK-irst Trento, Italy negri@fbk.eu Alessandro Marchetti CELCT Trento, Italy amarchetti@celct.it Yashar Mehdad FBK-irst Trento, Italy mehdad@fbk.eu Luisa Bentivogli FBK-irst Trento, Italy bentivo@fbk.eu Danilo Giampiccolo CELCT Trento, Italy giampiccolo@celct.it Abstract This paper presents the first round of the task on Cross-lingual Textual Entailment for Content Synchronization, organized within SemEval-2012. The task was designed to pro- mote research on semantic inference over texts written in different languages, targeting at the same time a real application scenario. Par- ticipants were presented with datasets for dif- ferent language pairs, where multi-directional entailment relations (“forward”, “backward”, “bidirectional”, “no entailment”) had to be identified. We report on the training and test data used for evaluation, the process of their creation, the participating systems (10 teams, 92 runs), the approaches adopted and the re- sults achieved. 1 Introduction The cross-lingual textual entailment task (Mehdad et al., 2010) addresses textual entailment (TE) recog- nition (Dagan and Glickman, 2004) under the new dimension of cross-linguality, and within the new challenging application scenario of content synchro- nization. Cross-linguality represents a dimension of the TE recognition problem that has been so far only par- tially investigated. The great potential for integrat- ing monolingual TE recognition components into NLP architectures has been reported in several ar- eas, including question answering, information re- trieval, information extraction, and document sum- marization. However, mainly due to the absence of cross-lingual textual entailment (CLTE) recognition components, similar improvements have not been achieved yet in any cross-lingual application. The CLTE task aims at prompting research to fill this gap. Along such direction, research can now ben- efit from recent advances in other fields, especially machine translation (MT), and the availability of: i) large amounts of parallel and comparable corpora in many languages, ii) open source software to com- pute word-alignments from parallel corpora, and iii) open source software to set up MT systems. We believe that all these resources can positively con- tribute to develop inference mechanisms for multi- lingual data. Content synchronization represents a challenging application scenario to test the capabilities of ad- vanced NLP systems. Given two documents about the same topic written in different languages (e.g. Wiki pages), the task consists of automatically de- tecting and resolving differences in the information they provide, in order to produce aligned, mutually enriched versions of the two documents. Towards this objective, a crucial requirement is to identify the information in one page that is either equivalent or novel (more informative) with respect to the content of the other. The task can be naturally cast as an entailment recognition problem, where bidirectional and unidirectional entailment judgments for two text fragments are respectively mapped into judgments about semantic equivalence and novelty. Alterna- tively, the task can be seen as a machine translation evaluation problem, where judgments about seman- tic equivalence and novelty depend on the possibility to …

Content truncated to 500 words