DOMLIN at SemEval-2019 Task 8: Automated Fact Checking exploiting Ratings in Community Question Answering Forums

Proceedings of the 13th International Workshop on Semantic Evaluation (SemEval-2019), pages 1149–1154 Minneapolis, Minnesota, USA, June 6–7, 2019. ©2019 Association for Computational Linguistics 1149 DOMLIN at SemEval-2019 Task 8: Automated Fact Checking exploiting Ratings in Community Question Answering Forums Dominik Stammbach DFKI, Saarbr¨ucken, Germany dominik.stammbach@dfki.de Stalin Varanasi DFKI, Saarbr¨ucken, Germany stalin.varanasi@dfki.de G¨unter Neumann DFKI, Saarbr¨ucken, Germany neumann@dfki.de Abstract In the following, we describe our system de- veloped for the Semeval2019 Task 8. We fine- tuned a BERT checkpoint on the qatar living forum dump and used this checkpoint to train a number of models. Our hand-in for subtask A consists of a fine-tuned classifier from this BERT checkpoint. For subtask B, we first have a classifier deciding whether a comment is fac- tual or non-factual. If it is factual, we retrieve intra-forum evidence and using this evidence, have a classifier deciding the comment’s ve- racity. We trained this classifier on ratings which we crawled from qatarliving.com. 1 Introduction This paper contains our system description for the SemEval2019 task 8 about Fact Checking in Com- munity Forums. The task 8 is divided into two subtasks: In subtask A, the goal is to determine whether a question asks for a factual answer, an opinion or is just posed to socialize. In subtask B, if we have a question asking for a factual an- swer, we classify the answers to such a question into three categories, namely the answer is either true, false or non-factual, i.e. it does not answer the question in a factual way. For subtask A, we trained a BERT classifier on the training set and optimized hyper-parameters on the development set. For subtask B, we de- cided to tackle the challenge with two binary clas- sifiers: Firstly, we decide whether a comment is factual or not. If our classifier decides that a com- ment is factual, we retrieve intra-forum evidence to determine the comment’s veracity using a tex- tual entailment approach. Given the small training set for subtask B, we decided to leverage openly available information on qatarliving.com to cre- ate a medium-sized training set. We found that comments on qatarliving.com are sometimes as- sociated with ratings1 (ranging from 1 to 5) and discovered that high ratings often correspond to replies answering the question in a true way. If a comment has recieved a low rating, we inferred that the comment was most likely not helpful to answer the question and therefore we decided to treat it as a false reply. 2 Related Work Automated Fact Checking is recently mostly per- ceived as a number of tasks which can be pipelined together. In the FEVER shared task, most par- ticipating systems would first find evidence and then train textual entailment models (Thorne et al., 2018). Related work for Fact Checking in com- munity forums considers a multi-faceted approach incorporating firstly what is said, how it is said and by whom and secondly external evidence from ei- ther the web or from the forum itself (Mihaylova et al., 2018). An …

Content truncated to 500 words