Knowing What to Believe (when you already know something)

Proceedings of the 23rd International Conference on Computational Linguistics (Coling 2010), pages 877–885, Beijing, August 2010 Knowing What to Believe (when you already know something) Jeff Pasternack Dan Roth University of Illinois, Urbana-Champaign {jpaster2, danr}@uiuc.edu Abstract Although much work in NLP has focused on simply determining what a document means, we also must know whether or not to believe it. Fact-finding algorithms at- tempt to identify the “truth” among com- peting claims in a corpus, but fail to take advantage of the user’s prior knowl- edge and presume that truth itself is uni- versal and objective rather than subjec- tive. We introduce a framework for incor- porating prior knowledge into any fact- finding algorithm, expressing both gen- eral “common-sense” reasoning and spe- cific facts already known to the user as first-order logic and translating this into a tractable linear program. As our results show, this approach scales well to even large problems, both reducing error and allowing the system to determine truth re- spective to the user rather than the major- ity. Additionally, we introduce three new fact-finding algorithms capable of outper- forming existing fact-finders in many of our experiments. 1 Introduction Although establishing the trustworthiness of the information presented to us has always been a challenge, the advent of the Information Age and the Internet has made it more critical. Blogs, wikis, message boards and other collaborative media have eliminated the high entry barrier– and, with it, the enforced journalistic standards–of older, established media such as newspapers and television, and even these sometimes loosen their fact-checking in the face of increased competitive pressure. Consequently, we find that corpora de- rived from these sources now offer far more nu- merous views of far more questionable veracity. If one author claims Mumbai is the largest city in the world, and another claims it is Seoul, who do we believe? One or both authors could be inten- tionally lying, honestly mistaken or, alternatively, of different viewpoints of what constitutes a “city” (the city proper? The metropolitan area?) Truth is not objective: there may be many valid definitions of “city”, but we should believe the claim that ac- cords with our user’s viewpoint. Note that the user may be another computational system rather than a human (e.g. building a knowledge base of city sizes for question answering), and often neither the user’s nor the information source’s perspective will be explicit (e.g. an author will not fully elabo- rate “the largest city by metropolitan area bounded by...”) but will instead be implied (e.g. a user’s statement that “I already know the population of city A is X, city B is Y...” implies that his defini- tion of a city accords with these figures). The most basic approach is to take a vote: if multiple claims are mutually exclusive of each other, select the one asserted by the most sources. In our experiments, sources will be the authors of the document containing the claim, but other sources could be publishers/websites (when no authorship is given), …

Content truncated to 500 words