Automated Fact-Checking of Claims from Wikipedia
13321Authors
nan
Abstract
AbstractAutomated fact checking is becoming increasingly vital as both truthful and fallacious information accumulate online. Research on fact checking has benefited from large-scale datasets such as FEVER and SNLI. However, such datasets suffer from limited applicability due to the synthetic nature of claims and/or evidence written by annotators that differ from real claims and evidence on the internet. To this end, we present WikiFactCheck-English, a dataset of 124k+ triples consisting of a claim, context and an evidence document extracted from English Wikipedia articles and citations, as well as 34k+ manually written claims that are refuted by the evidence documents. This is the largest fact checking dataset consisting of real claims and evidence to date; it will allow the development of fact checking systems that can better process claims and evidence in the real world. We also show that for the NLI subtask, a logistic regression system trained using existing and novel features achieves peak accuracy of 68%, providing a competitive baseline for future work. Also, a decomposable attention model trained on SNLI significantly underperforms the models trained on this dataset, suggesting that models trained on manually generated data may not be sufficiently generalizable or suitable for fact checking real-world claims.
Keywords
Full Content
Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 668–675 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 668 Multimodal corpus of bidirectional conversation of human-human and human-robot interaction during fMRI scanning Birgit Rauchbauer 1,2,3, Youssef Hmamouche2,4, Brigitte Bigi2, Laurent Prévot2, Magalie Ochs5, Thierry Chaminade1 1Aix Marseille Université, CNRS, INT, Marseille, France 2Aix Marseille Université, CNRS, LPL, Aix-en-Provence, France 3Aix Marseille Université, CNRS, LNC, Marseille, France 4Aix Marseille Université, CNRS, LIS, Marseille, France 5 Aix Marseille Univ, Université de Toulon, CNRS, LIS, Marseille, France birgit.rauchbauer@univ-amu.fr; youssef.hmamouche@lis-lab.fr; laurent.prevot@univ-amu.fr; magalie.ochs@lis- lab.fr; thierry.chaminade@univ-amu.fr Abstract In this paper we present investigation of real-life, bi-directional conversations. We introduce the multimodal corpus derived from these natural conversations alternating between human-human and human-robot interactions. The human-robot interactions were used as a control condition for the social nature of the human-human conversations. The experimental set up consisted of conversations between the participant in a functional magnetic resonance imaging (fMRI) scanner and a human confederate or conversational robot outside the scanner room, connected via bidirectional audio and unidirectional videoconferencing (from the outside to inside the scanner). A cover story provided a framework for natural, real-life conversations about images of an advertisement campaign. During the conversations we collected a multimodal corpus for a comprehensive characterization of bi-directional conversations. In this paper we introduce this multimodal corpus which includes neural data from functional magnetic resonance imaging (fMRI), physiological data (blood flow pulse and respiration), transcribed conversational data, as well as face and eye-tracking recordings. Thus, we present a unique corpus to study human conversations including neural, physiological and behavioral data. Keywords: conversation, physiology, artificial agents 1. Introduction Reciprocal interactions with others allow humans to establish and maintain social bonds. We investigated human interaction using the most ubiquitous form of communication through language in conversation. Our objective was to characterize conversations through a multimodal approach for a comprehensive investigation of human interaction. As a control condition we used interaction with a conversational robot. This allowed us to change the social nature of the interlocutor, while preserving the reciprocal dynamics of social exchange. The social nature of the interaction with human derives from the assumption that humans adopt an intentional stance towards other humans, but not artificial agents, such as conversational robots. Adopting an intentional stance assumes that the interaction partner is capable of having mental states (Dennett, 1989). This allows the ascription of intentions and beliefs to explain the behavior of interaction partner (i.e., mentalizing (Frith & Frith, 1999)). We thus hypothesized that interaction with a human, as compared to a robot, would elicit both mentalizing mechanisms and the motivation to establish a relationship (i.e. social motivation). To investigate this, we collected multimodal data including neuro-physiological responses of functional magnetic resonance imagining (fMRI), as well as physiological data and observable behaviors such as gaze, facial expressions and verbal productions during conversation. All these modalities have been shown sensitive to social and emotional engagement in interactions. We compared conversations between participants and another human …
Content truncated to 500 words
Labeled Text
AbstractAutomated fact checking is becoming increasingly vital as both truthful and fallacious information accumulate online. Research on fact checking has benefited from large-scale datasets such as FEVER and SNLI. However, such datasets suffer from limited applicability due to the synthetic nature of claims and/or evidence written by annotators that differ from real claims and evidence on the internet. To this end, we present WikiFactCheck-English, a dataset of 124k+ triples consisting of a claim, context and an evidence document extracted from English Wikipedia articles and citations, as well as 34k+ manually written claims that are refuted by the evidence documents. This is the largest fact checking dataset consisting of real claims and evidence to date; it will allow the development of fact checking systems that can better process claims and evidence in the real world. We also show that for the NLI subtask, a logistic regression system trained using existing and novel features achieves peak accuracy of 68%, providing a competitive baseline for future work. Also, a decomposable attention model trained on SNLI significantly underperforms the models trained on this dataset, suggesting that models trained on manually generated data may not be sufficiently generalizable or suitable for fact checking real-world claims.
Publication Details
- Paper ID:
- 13321
- Year:
- 2020
- Publication:
- TIPSTER
- Publication Type:
- conf
- DOI:
- nan