Identification and Verification of Simple Claims about Statistical Properties

Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 2596–2601, Lisbon, Portugal, 17-21 September 2015. c⃝2015 Association for Computational Linguistics. Identification and Verification of Simple Claims about Statistical Properties Andreas Vlachos and Sebastian Riedel Department of Computer Science University College London {a.vlachos,s.riedel}@cs.ucl.ac.uk Abstract In this paper we study the identification and verification of simple claims about statistical properties, e.g. claims about the population or the inflation rate of a coun- try. We show that this problem is similar to extracting numerical information from text and following recent work, instead of annotating data for each property of inter- est in order to learn supervised models, we develop a distantly supervised base- line approach using a knowledge base and raw text. In experiments on 16 statistical properties about countries from Freebase we show that our approach identifies sim- ple statistical claims about properties with 60% precision, while it is able to verify these claims without requiring any explicit supervision for either tasks. Furthermore, we evaluate our approach as a statistical property extractor and we show it achieves 0.11 mean absolute percentage error. 1 Introduction Statistical properties are commonly used to de- scribe entities, e.g. population for countries, net value for companies, points scored for ath- letes, etc. Claims about such properties are very common in news articles and social media, how- ever they can be erroneous, either due to au- thor error or negligence at the time of writing or because they eventually become out of date. While manual verification (also referred to as fact- checking) is conducted by journalists in news or- ganizations and dedicated websites such as www. emergent.info, the volume of the claims calls for automated approaches, which is one of the main objectives of computational journalism (Co- hen et al., 2011; Flew et al., 2012). In this paper we develop a baseline approach to identify and verify simple claims about statistical Text: Lesotho, a landlocked enclave of South Africa, has a population of nearly 2 million and covers an area slightly smaller than the U.S. state of Maryland. Entity: Lesotho Property: population Value claimed in text: 2,000,000 Value in knowledge base: 2,193,843 Absolute percentage error: 0.09 Figure 1: Claim identification and verification. properties against a database. The task is illus- trated in Figure 1. Given a sentence, we first iden- tify whether it contains a claim about a property we are interested in (population in the exam- ple), which entity it is about and the value claimed (Lesotho and 2,000,000 respectively). We then proceed to verify the value claimed in text for the property of this entity against the value known in a knowledge base such as Freebase and return a score reflecting the accuracy of the claim (abso- lute percentage error in the example). Claim identification is essentially an instance of information extraction. While it would be possi- ble to develop supervised models, this would re- quire expensive manual data annotation for each property of interest. Instead, we follow the dis- tant supervision …

Content truncated to 500 words