Divide and Conquer: Crowdsourcing the Creation of Cross-Lingual Textual Entailment Corpora

Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, pages 670–679, Edinburgh, Scotland, UK, July 27–31, 2011. c⃝2011 Association for Computational Linguistics Divide and Conquer: Crowdsourcing the Creation of Cross-Lingual Textual Entailment Corpora Matteo Negri FBK-irst Trento, Italy negri@fbk.eu Luisa Bentivogli FBK-irst Trento, Italy bentivo@fbk.eu Yashar Mehdad FBK-irst and University of Trento Trento, Italy mehdad@fbk.eu Danilo Giampiccolo CELCT Trento, Italy giampiccolo@celct.it Alessandro Marchetti CELCT Trento, Italy amarchetti@celct.it Abstract We address the creation of cross-lingual tex- tual entailment corpora by means of crowd- sourcing. Our goal is to define a cheap and replicable data collection methodology that minimizes the manual work done by expert annotators, without resorting to preprocess- ing tools or already annotated monolingual datasets. In line with recent works empha- sizing the need of large-scale annotation ef- forts for textual entailment, our work aims to: i) tackle the scarcity of data available to train and evaluate systems, and ii) promote the re- course to crowdsourcing as an effective way to reduce the costs of data collection without sacrificing quality. We show that a complex data creation task, for which even experts usu- ally feature low agreement scores, can be ef- fectively decomposed into simple subtasks as- signed to non-expert annotators. The resulting dataset, obtained from a pipeline of different jobs routed to Amazon Mechanical Turk, con- tains more than 1,600 aligned pairs for each combination of texts-hypotheses in English, Italian and German. 1 Introduction Cross-lingual Textual Entailment (CLTE) has been recently proposed by (Mehdad et al., 2010; Mehdad et al., 2011) as an extension of Textual Entailment (Dagan and Glickman, 2004). The task consists of deciding, given a text (T) and an hypothesis (H) in different languages, if the meaning of H can be in- ferred from the meaning of T. As in other NLP appli- cations, both for monolingual and cross-lingual TE, the availability of large quantities of annotated data is an enabling factor for systems development and evaluation. Until now, however, the scarcity of such data on the one hand, and the costs of creating new datasets of reasonable size on the other, have repre- sented a bottleneck for a steady advancement of the state of the art. In the last few years, monolingual TE corpora for English and other European languages have been created and distributed in the framework of sev- eral evaluation campaigns, including the RTE Chal- lenge1, the Answer Validation Exercise at CLEF2, and the Textual Entailment task at EVALITA3. De- spite the differences in the design of the tasks, all the released datasets were collected through simi- lar procedures, always involving expensive manual work done by expert annotators. Moreover, in the data creation process, large amounts of hand-crafted T-H pairs often have to be discarded in order to re- tain only those featuring full agreement, in terms of the assigned entailment judgements, among multiple annotators. The amount of discarded pairs is usually high, contributing to increase the costs of creating textual entailment datasets4. The issues related to the shortage of …

Content truncated to 500 words