SolomonLab at SemEval-2019 Task 8: Question Factuality and Answer Veracity Prediction in Community Forums

Proceedings of the 13th International Workshop on Semantic Evaluation (SemEval-2019), pages 1165–1171 Minneapolis, Minnesota, USA, June 6–7, 2019. ©2019 Association for Computational Linguistics 1165 SolomonLab at SemEval-2019 Task 8: Question Factuality and Answer Veracity Prediction in Community Forums Sudeep Kumar Sahoo∗ Rohit R.R Ankita Gupta∗ Vertika Srivastava Divya Prakash Yeon Hyang Kim Samsung R&D Institute India, Bangalore {gupta.ankita, sudeep.sahoo, p.divya, rohit.r.r, v.srivastava, purine.kim}@samsung.com Abstract We describe our system for SemEval-2019, Task 8 on “Fact-Checking in Community Question Answering Forums (cQA)”. cQA forums are very prevalent nowadays, as they provide an effective means for communities to share knowledge. Unfortunately, this shared information is not always factual and fact- verified. In this task, we aim to identify fac- tual questions posted on cQA and verify the veracity of answers to these questions. Our approach relies on data augmentation and ag- gregates cues from several dimensions such as semantics, linguistics, syntax, writing style and evidence obtained from trusted external sources. In subtask A, our submission is ranked 3rd, with an accuracy of 83.14%. Our current best solution stands 1st on the leader- board with 88% accuracy. In subtask B, our present solution is ranked 2nd, with 58.33% MAP score. 1 Introduction With the rising popularity of online commu- nity question answering (cQA) systems such as Quora, StackOverflow, and Qatar Living forum (QLF), the amount of information shared over these platforms is also increasing rapidly with time. These forums provide effective information sharing mechanism to their users who can seek an- swers to their queries as well as post answers to the questions. However, the information shared on such platforms may not always be factual and cor- rect. The responders may misunderstand the ques- tion being asked or merely ignore certain specific details. At times, the information shared may even be false or ambiguous in the desired context. This is aggravated by lack of moderation and system- atic control on cQA forums. The Semeval-2019 Task 81 on “Fact Checking in Community Ques- ∗*Equal Contribution 1http://alt.qcri.org/semeval2019/index.php?id=tasks tion Answering Forums” aims to solve this real- life problem. The above task tries to explore the veracity of an answers to a question posted on QLF. While the precedent tasks such as SemEval (Nakov et al., 2015, 2016, 2017), address the issue of ranking answers according to their relevance to a question, the task-at-hand is the first one to consider the cor- rectness of answers. This task is formulated as a two-stage problem. The first stage aims to iden- tify the user posts asking for factual information. The answers to the identified factual questions are then fact-verified in the second stage. Both the subtasks are designed as 3-class supervised clas- sification problems. More specifically, the first stage or subtask A addresses the problem of determining whether the posted question asks for factual information, an opinion/advice or is just meant for socializing. For example, “what is Ooredoo customer service number?” asks for factual information, whereas “What was your first car?” is socializing and “which is the best bank around?” …

Content truncated to 500 words