AUTOHOME-ORCA at SemEval-2019 Task 8: Application of BERT for Fact-Checking in Community Forums

Proceedings of the 13th International Workshop on Semantic Evaluation (SemEval-2019), pages 870–876 Minneapolis, Minnesota, USA, June 6–7, 2019. ©2019 Association for Computational Linguistics 870 AUTOHOME-ORCA at SemEval-2019 Task 8: Application of BERT for Fact-Checking in Community Forums Zhengwei Lv1 Duoxing Liu1 Haifeng Sun2 Xiao Liang1 Tao Lei1 Zhizhong Shi1 Feng Zhu1 Lei Yang1 1 Autohome Inc., Beijing, China 2 Beijing University of Posts and Telecommunications, Beijing, China {lvzhengwei,liuduoxing,liangxiao12030,leitao,shizhizhong,zhufeng,yanglei}@autohome.com.cn hfsun@bupt.edu.cn Abstract Fact checking is an important task for main- taining high quality posts and improving user experience in Community Question Answer- ing forums. Therefore, the SemEval-2019 task 8 is aimed to identify factual question (subtask A) and detect true factual information from corresponding answers (subtask B). In order to address this task, we propose a system based on the BERT model with meta information of questions. For the subtask A, the outputs of fine-tuned BERT classification model are combined with the feature of length of ques- tions to boost the performance. For the subtask B, the predictions of several variants of BERT model encoding the meta information are com- bined to create an ensemble model. Our sys- tem achieved competitive results with an ac- curacy of 0.82 in the subtask A and 0.83 in the subtask B. The experimental results vali- date the effectiveness of our system. 1 Introduction The Community Question Answering (CQA) fo- rums are gaining more and more popularity be- cause they can offer great opportunity for users to get appropriate answers to their questions from other users. Meanwhile, the accumulated mas- sive questions and answers in CQA forums present a new challenge to provide valuable information for users more effectively. Therefore, researchers have shown an increased interest in CQA systems (Srba and Bielikova, 2016; Wang et al., 2018), aiming to facilitate efficient knowledge acquisi- tion and circulation. Specifically, a large por- tion of researches mainly focus on the two tasks: find relevant questions to a new question to reuse corresponding answers (Question Retrieval), and search for relevant answers among existing an- swers to other questions (Answer Selection). Despite a great deal of research on CQA, there are relatively few studies focusing on the quality of questions and answers. Actually, the credibility of answers is an important aspect, which can di- rectly affect the user experience for CQA forums. In order to check the veracity of answers automat- ically, some recent works (Karadzhov et al., 2017; Mihaylova et al., 2018) attempt to utilize external sources and extract appropriate features for classi- fication. Considering the importance of informa- tion veracity in CQA forums, the fact checking of answers is still an issue that is worth investigating further. Therefore, the SemEval-2019 task 8 aims to conduct fact checking in CQA forums. In order to detect the veracity of answers, it is necessary to identify whether the questions are factual firstly. The task is comprised of two subtasks: the subtask A is targeted to identify whether a question is ask- ing for factual information, an opinion/advice or socializing. Given factual questions, …

Content truncated to 500 words