AnswerFact: Fact Checking in Product Question Answering

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 2407–2417, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 2407 AnswerFact: Fact Checking in Product Question Answering∗ Wenxuan Zhang†, Yang Deng†, Jing Ma‡ and Wai Lam† †The Chinese University of Hong Kong ‡Hong Kong Baptist University {wxzhang,ydeng,wlam}@se.cuhk.edu.hk majing@comp.hkbu.edu.hk Abstract Product-related question answering platforms nowadays are widely employed in many E- commerce sites, providing a convenient way for potential customers to address their con- cerns during online shopping. However, the misinformation in the answers on those plat- forms poses unprecedented challenges for users to obtain reliable and truthful product in- formation, which may even cause a commer- cial loss in E-commerce business. To tackle this issue, we investigate to predict the veracity of answers in this paper and introduce Answer- Fact, a large scale fact checking dataset from product question answering forums. Each answer is accompanied by its veracity label and associated evidence sentences, providing a valuable testbed for evidence-based fact check- ing tasks in QA settings. We further propose a novel neural model with tailored evidence ranking components to handle the concerned answer veracity prediction problem. Extensive experiments are conducted with our proposed model and various existing fact checking meth- ods, showing that our method outperforms all baselines on this task. 1 Introduction The ability to ask questions during online shopping is found to be a key factor for customers to make purchase decisions (Smith and Anderson, 2016). To this end, product-related community question answering (PQA) platforms have emerged in many E-commerce sites such as Amazon and Taobao, al- lowing users to pose their concerns as questions and receive answers from fellow users to obtain use- ful product information. However, similar to other community question answering (CQA) platforms, ∗ The work described in this paper is substantially sup- ported by a grant from the Research Grant Council of the Hong Kong Special Administrative Region, China (Project Code: 14200719). Question: Is this egg coker automatic shut off? Answer (Claim): Yes there’s an automatic shut-off when the cooking cycle is finished. (Verdict : FALSE) Evidence: s1: A buzzer sounds to let you know the eggs are done, I wish it would just shut off instead. s2: When the alarm sounds you need to turn it off and open it. s3: I would have liked the cooker to turn off automatically but instead a bell rings until you turn if off. s4: Also, by the time the timer goes off, the hot pan has a burning smell. s5: And it turns off itself after the bell rings. . . . Table 1: An example instance in AnswerFact, where the answer is the claim to be verified. The relevant prod- uct information are provided as evidence sentences. the user-provided answers on PQA platforms vary significantly on their qualities (Zhang et al., 2020b), and more seriously, their veracity due to the lack of systematic quality control (Mihaylova et al., 2018). Those untruthful answers may attribute to multiple factors such as misunderstandings of …

Content truncated to 500 words