Assessing the Verifiability of Attributions in News Text

Proceedings of the The 8th International Joint Conference on Natural Language Processing, pages 754–763, Taipei, Taiwan, November 27 – December 1, 2017 c⃝2017 AFNLP Assessing the Verifiability of Attributions in News Text Edward Newell McGill University Computer Science Ariane Schang McGill University Computer Science Drew Margolin Cornell University Dept. of Communication Derek Ruths McGill University Computer Science {edward.newell, ariane.schang}@mail.mcgill.ca, dm658@cornell.edu, derek.ruths@mcgill.ca Abstract When reporting the news, journalists rely on the statements of stakeholders, experts, and officials. The attribution of such a statement is verifiable if its fidelity to the source can be confirmed or denied. In this paper, we develop a new NLP task: determining the verifiability of an attribution based on linguistic cues. We operationalize the notion of verifiability as a score between 0 and 1 using hu- man judgments in a comparison-based ap- proach. Using crowdsourcing, we cre- ate a dataset of verifiability-scored attri- butions, and demonstrate a model that achieves an RMSE1 of 0.057 and Spear- man’s rank correlation of 0.95 to human- generated scores. We discuss the appli- cation of this technique to the analysis of mass media. 1 Introduction An attribution occurs when an author or speaker represents the discourse, attitude, or inner state of an external source (Piazza, 2009). Attributions are found in virtually every genre of discourse (Fair- clough, 1995), but are fundamental to news report- ing, where attribution to credible sources is a basic feature of objective, unbiased “hard news” (Esser and Umbricht, 2014). Recently news media have come under increasing scrutiny for spreading bi- ased and even fabricated information2. This trend suggests the need for scalable, computational ap- proaches to understanding attribution. From a natural language processing perspec- tive, attribution is a fundamental phenomenon that 1Root mean squared error 2theguardian.com/media/2016/dec/18/what-is-fake- news-pizzagate touches a broad set of applications, including sum- marization, question answering, information ex- traction, and discourse analysis. Once content is scoped under attribution, its contribution to the discourse can change substantially depending on the source of the attribution and their relationship to the statement. For instance, in 2001 U.S. Pres- ident George W. Bush famously warned that in fighting terrorists, nations were either “with us or against us”3. This statement was threatening not only because it was made by the president, or be- cause it was blunt, but because such blunt state- ments are not normally made by national leaders. If they are to reach human-level performance, sys- tems for automated text understanding must not only accurately segment attribution in the flow of text, but also represent the many ways that attribu- tions can differ rhetorically. One important way in which attributions can differ, particularly with respect to news reporting, is in their verifiability, the ease with which an at- tribution’s fidelity to the source can be checked. Consider the following hypothetical attributions: Lindsay Walls, CEO of Inovatron, said in a press release yesterday that the “al- legations of intentionally selling sub-par product are completely unfounded.” A source close to the issue hinted that quality control standards had been on …

Content truncated to 500 words