Automated Historical Fact-Checking by Passage Retrieval, Word Statistics, and Virtual Question-Answering

Proceedings of the The 8th International Joint Conference on Natural Language Processing, pages 967–975, Taipei, Taiwan, November 27 – December 1, 2017 c⃝2017 AFNLP Automated Historical Fact-Checking by Passage Retrieval, Word Statistics, and Virtual Question-Answering Mio Kobayashi1 Ai Ishii1 Chikara Hoshino1 Hiroshi Miyashita1 Takuya Matsuzaki2 1Nihon Unisys Ltd., Japan {mio.kobayashi, ai.ishii, chikara.hoshino, hiroshi.miyashita}@unisys.co.jp 2Nagoya University, Japan matuzaki@nuee.nagoya-u.ac.jp Abstract This paper presents a hybrid approach to the verification of statements about his- torical facts. The test data was collected from the world history examinations in a standardized achievement test for high school students. The data includes var- ious kinds of false statements that were carefully written so as to deceive the stu- dents while they can be disproven on the basis of the teaching materials. Our sys- tem predicts the truth or falsehood of a statement based on text search, word cooc- currence statistics, factoid-style question answering, and temporal relation recog- nition. These features contribute to the judgement complementarily and achieved the state-of-the-art accuracy. 1 Introduction The proliferation of social media in the Internet drastically changed the status of traditional jour- nalism, which has been an indispensable build- ing block of modern democracy. News are now produced, propagated, and consumed by people in quite a different way than twenty years ago (Pew Research Center, 2016). The downside is that fake news and hoaxes spread through the social net- work as quickly as those from trustable sources. A mechanism for fact-checking, i.e., finding a sup- port or disproof of a claim in a credible informa- tion source, is thus needed as a new social infras- tructure. The sheer amount of the information flow as well as the decentralized nature of the social me- dia calls for support to the fact-checking by infor- mation technology (Cohen et al., 2011a,b). Al- though its full automation seems to be beyond cur- rent technology (Vlachos and Riedel, 2014; Has- Context: ... During the period of the Carolingian dynasty of Francia, the Roman Catholic Church preached that the religious cleansing of sins was necessary in order to achieve salvation after death. ... Instruction: From (1)-(4) below, choose the one correct sentence concerning events during the 8th century when the kingdom referred to in the underlined portion was established. Choices: (1) Pepin destroyed the Kingdom of the Lombards. (2) Charlemagne repelled the Magyars. (3) The reign of Emperor Taizong of Tang was called the Kaiyuan era. (4) The reign of Harun al-Rashid began. Figure 1: Example of a True-or-False question san et al., 2015), even its partial automation would greatly enhance the power of the current fact- checking services. As a step towards this direction, we take up the automatic verification of a statement about histori- cal facts against credible information sources. The test statements are collected from the world his- tory examinations in a standardized achievement test for high school students in Japan (the National Center Test for University Admissions, NTCUA). Approximately 60% of the NCTUA world history exams are “True-or-False” questions. A question in this format …

Content truncated to 500 words