Monolingual Social Media Datasets for Detecting Contradiction and Entailment

Monolingual Social Media Datasets for Detecting Contradiction and Entailment Piroska Lendvai*, Isabelle Augenstein**, Kalina Bontcheva**, Thierry Declerck* *Dept. of Computational Linguistics, Saarland University, Germany **Dept. of Computer Science, University of Sheffield, GB piroska.r@gmail.com, {i.augenstein,k.bontcheva}@sheffield.ac.uk, declerck@dfki.de Abstract Entailment recognition approaches are useful for application domains such as information extraction, question answering or summari- sation, for which evidence from multiple sentences needs to be combined. We report on a new 3-way judgement Recognizing Textual Entailment (RTE) resource that originates in the Social Media domain, and explain our semi-automatic creation method for the special purpose of information verification, which draws on manually established rumourous claims reported during crisis events. From about 500 English tweets related to 70 unique claims we compile and evaluate 5.4k RTE pairs, while continue automatizing the workflow to generate similar-sized datasets in other languages. Keywords: textual entailment, social media, verification 1. Introduction In this paper, we report on building a special-purpose Rec- ognizing Textual Entailment (RTE) dataset in the context of information verification in user-generated content (Men- doza et al., 2010; Qazvinian et al., 2011; Procter et al., 2013) for the PHEME project1. The dataset is compiled based on naturally occurring contradiction in manually la- beled claims in crisis events discussed on Twitter, and to our knowledge is the first resource for RTE in the social media and verification domain. The detection of semantic inference phenomena between natural language text snippets, such as contradiction, en- tailment, and stance, is targeted by a number of research communities. Its most focused interest group formalizes in- ference tasks in the generic framework of RTE2. RTE is ap- plied to benefit several Natural Language Processing (NLP) tasks, such as information retrieval or text summarization. The task of RTE is to recognise the relationship between sentence pairs, specifically if they entail or contradict each other, or neither of those. A bottleneck for this task is obtaining training data. The creation of natural language data annotated for inference phenomena is so far a nontrivial and largely manual pro- cedure, yielding expensive resources that are nonetheless problematically portable to new text genres and application domains. Existing initiatives have often created RTE data by syntactic and lexical transformations with predictable effects asking annotators to (re)write sentences taken from gold standards for other tasks such as question answer- ing (Bar-Haim et al., 2006) and image and video descrip- tion (Bowman et al., 2015; Marelli et al., 2014). RTE tasks may involve 2-way or 3-way inference judge- ments. In case of a 2-way judgement, the class to guess is either Entailment or Nonentailment. On the 3-way judge- ment scheme the Nonentailment class is further differen- tiated into Contradiction and Unknown. The presence of contradictory statements in social media can be indicative 1http://www.pheme.eu/ 2http://www.aclweb.org/aclwiki/index.php?title=Recognizing Textual Entailment for mis-/disinformation, controversy or speculation, which are important triggers in veracity checking procedures. Our present contribution therefore addresses the transformation of a project-internal corpus of annotated microblog texts into a 3-way RTE dataset. Contradiction is the RTE relation when two texts falsify one another’s …

Content truncated to 500 words