Evidence-based Trustworthiness

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 413–423 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 413 Evidence-based Trustworthiness Yi Zhang Zachary G. Ives Department of Computer and Information Science University of Pennsylvania {yizhang5, zives, danroth}@cis.upenn.edu Dan Roth Abstract The information revolution brought with it information pollution. Information retrieval and extraction help us cope with abundant in- formation from diverse sources. But some sources are of anonymous authorship, and some are of uncertain accuracy, so how can we determine what we should actually be- lieve? Not all information sources are equally trustworthy, and simply accepting the majority view is often wrong. This paper develops a general framework for estimating the trustworthiness of information sources in an environment where multiple sources provide claims and supporting evi- dence, and each claim can potentially be pro- duced by multiple sources. We consider two settings: one in which information sources directly assert claims, and a more realistic and challenging one, in which claims are in- ferred from evidence provided by sources, via (possibly noisy) NLP techniques. Our key contribution is to develop a family of proba- bilistic models that jointly estimate the trust- worthiness of sources, and the credibility of claims they assert. This is done while ac- counting for the (possibly noisy) NLP needed to infer claims from evidence supplied by sources. We evaluate our framework on sev- eral datasets, showing strong results and sig- nificant improvement over baselines. 1 Introduction The emergence of social networks and news ag- gregators — combined with ill-informed posts, deliberate efforts to create and spread sensation- alized information, and a strongly polarized po- litical environment — makes it very difficult to establish what is really known. Therefore, fact checking seeks to assess whether the claim is true or false, or to provide a confidence level for the claim given textual evidence (Hassan et al., 2017; Wang, 2017; Wang et al., 2018). A typical fact checking pipeline consists of document retrieval, sentence-level evidence selection, and textual en- tailment stages (Thorne et al., 2018). However, this pipeline is local in that it applies to a given claim. The missing step here is to assess the trust- worthiness of the sources producing the claims and evidence. This is a global step that, in princi- ple, accounts for all claims made by a source and all sources making a claim. Previous work has studied how to estimate the trustworthiness or credibility of information sources for fact-finding (Vydiswaran et al.; Paster- nack and Roth, 2013), truth discovery (Dong et al.; Pochampally et al., 2014; Dong et al., 2015; Li et al., 2016) and crowdsourcing (Sabou et al., 2012; Hovy et al., 2013; Gao et al., 2015). Usu- ally, given a list of conflicting facts, e.g. “source s asserts claim c”, or “annotator x labels data item t by label y”, we detect the true claims or correct labels for the data item by resolving conflicts, and then compute the trustworthiness of …

Content truncated to 500 words