Language-Aware Truth Assessment of Fact Candidates

Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics, pages 1009–1019, Baltimore, Maryland, USA, June 23-25 2014. c⃝2014 Association for Computational Linguistics Language-Aware Truth Assessment of Fact Candidates Ndapandula Nakashole Carnegie Mellon University 5000 Forbes Avenue Pittsburgh, PA, 15213 ndapa@cs.cmu.edu Tom M. Mitchell Carnegie Mellon University 5000 Forbes Avenue Pittsburgh, PA, 15213 tom.mitchell@cs.cmu.edu Abstract This paper introduces FactChecker, language-aware approach to truth-finding. FactChecker differs from prior approaches in that it does not rely on iterative peer voting, instead it leverages language to infer believability of fact candidates. In particular, FactChecker makes use of lin- guistic features to detect if a given source objectively states facts or is speculative and opinionated. To ensure that fact candidates mentioned in similar sources have similar believability, FactChecker augments objectivity with a co-mention score to compute the overall believability score of a fact candidate. Our experiments on various datasets show that FactChecker yields higher accuracy than existing approaches. 1 Introduction Truth-finding algorithms aim to separate true statements (facts) from false information. More specifically, given a set of statements whose truth- fulness is unknown (fact candidates), the key goal of truth-finding algorithms is to generate a ranking such that true statements are ranked ahead of false ones. Truth-finders have the potential to address a major obstacle on the Web: the problem of sources spreading inaccurate and conflicting information. This problem continues to grow with the develop- ment of tools for easy Web authorship. Blogs, fo- rums and social networking websites are not sub- ject to traditional journalistic standards. Conse- quently, the accuracy of information reported by these sources is often unclear. Even more estab- lished newspapers and websites may sometimes report false information as they race to break sto- ries. Therefore, truth-finding is becoming an in- creasingly important problem. Information extrac- tion projects aim to distill relational facts from nat- ural language text (Auer et al., 2007; Bollacker et al., 2008; Carlson et al., 2010; Fader et al., 2011; Nakashole et al., 2011; Del Corro and Gemulla, 2013). These projects have produced knowledge bases containing many millions of relational facts between entities. However, despite these impres- sive advances, there are still major limitations re- garding precision. Within the context of informa- tion extraction, fact extractors assign confidence scores to extracted facts. However, such scores are often tied to the extractor’s ability to read and understand natural language text. This is differ- ent from a score that indicates the degree to which a given fact candidate is believable. Such a be- lievability score is sometimes also referred to as a credibility score or truthfulness score. The be- lievability score reflects the likelihood that a given statement is true. Truth-finding algorithms aim to compute this score for each fact candidate. Prior truth-finding methods are mostly based on iterative voting, where votes are propagated from sources to fact candidates and then back to sources (Yin et al., 2007; Galland et al., 2010; Paster- nack and Roth, 2010; Li et al., 2011; Yin and Tan, 2011). At …

Content truncated to 500 words