SemEval-2019 Task 8: Fact Checking in Community Question Answering Forums

Proceedings of the 13th International Workshop on Semantic Evaluation (SemEval-2019), pages 860–869 Minneapolis, Minnesota, USA, June 6–7, 2019. ©2019 Association for Computational Linguistics 860 SemEval-2019 Task 8: Fact Checking in Community Question Answering Forums Tsvetomila Mihaylova,1 Georgi Karadzhov,2 Pepa Atanasova,3 Ramy Baly,4 Mitra Mohtarami,4 Preslav Nakov5 1 Instituto de Telecomunicac¸˜oes, Lisbon, Portugal, 2 SiteGround Hosting EOOD, Bulgaria 3 University of Copenhagen, Denmark 4 MIT Computer Science and Artificial Intelligence Laboratory, Cambridge, MA 5 Qatar Computing Research Institute, HBKU {tsvetomila.mihaylova, georgi.m.karadjov}@gmail.com, pepa@di.ku.dk, {baly,mitram}@mit.edu, pnakov@qf.org.qa Abstract We present SemEval-2019 Task 8 on Fact Checking in Community Question Answering Forums, which features two subtasks. Subtask A is about deciding whether a question asks for factual information vs. an opinion/advice vs. just socializing. Subtask B asks to pre- dict whether an answer to a factual question is true, false or not a proper answer. We re- ceived 17 official submissions for subtask A and 11 official submissions for Subtask B. For subtask A, all systems improved over the ma- jority class baseline. For Subtask B, all sys- tems were below a majority class baseline, but several systems were very close to it. The leaderboard and the data from the competition can be found at http://competitions. codalab.org/competitions/20022. 1 Overview The current coverage of the political landscape in both the press and in social media has led to an unprecedented situation. Like never before, a statement in an interview, a press release, a blog note, or a tweet can spread almost instanta- neously. The speed of proliferation leaves little time for double-checking claims against the facts, which has proven critical in politics, e.g., during the 2016 presidential campaign in the USA, which was dominated by fake news in social media and by false claims. Investigative journalists and volunteers have been working hard to get to the root of a claim and to present solid evidence in favor or against it. Manual fact-checking is very time-consuming, and thus automatic methods have been proposed to speed-up the process, e.g., there has been work on checking the factuality/credibility of a claim, of a news article, or of an information source (Ba et al., 2016; Zubiaga et al., 2016; Ma et al., 2016; Castillo et al., 2011; Baly et al., 2018). The process starts when a document is made pub- lic. First, an intrinsic analysis is carried out in which check-worthy text fragments are identified. Then, other documents that might support or re- but a claim in the document are retrieved from various sources. Finally, by comparing a claim against the retrieved evidence, a system can deter- mine whether the claim is likely true or likely false (or unsure, if no strong enough evidence either way could be found). For instance, Ciampaglia et al. (2015) do this using a knowledge graph de- rived from Wikipedia. The outcome could then be presented to a human expert for final judgement.1 For our two subtasks, we explore factuality in the context of Community Question Answering (cQA) forums. Forums such as StackOverflow, Yahoo! Answers, …

Content truncated to 500 words