Predicting Sentence-Level Factuality of News and Bias of Media Outlets

Proceedings of Recent Advances in Natural Language Processing, pages 1197–1206 Varna, Sep 4–6, 2023 https://doi.org/10.26615/978-954-452-092-2_127 1197 Predicting Sentence-Level Factuality of News and Bias of Media Outlets Francielle Vargas1,2, Kokil Jaidka3, Thiago A. S. Pardo1, Fabr´ıcio Benevenuto2 1Institute of Mathematical and Computer Sciences, University of S˜ao Paulo, Brazil 2Computer Science Department, Federal University of Minas Gerais, Brazil 3Centre for Trusted Internet and Community, National University of Singapore, Singapore francielleavargas@usp.br, jaidka@nus.edu.sg taspardo@icmc.usp.br, fabricio@dcc.ufmg.br Abstract Automated news credibility and fact-checking at scale require accurate prediction of news fac- tuality and media bias. This paper introduces a large sentence-level dataset, titled FactNews1, composed of 6,191 sentences expertly anno- tated according to factuality and media bias definitions proposed by AllSides2. We use Fact- News to assess the overall reliability of news sources by formulating two text classification problems for predicting sentence-level factu- ality of news reporting and bias of media out- lets. Our experiments demonstrate that biased sentences present a higher number of words compared to factual sentences, besides having a predominance of emotions. Hence, the fine- grained analysis of subjectivity and impartiality of news articles showed promising results for predicting the reliability of the entire media out- let. Finally, due to the severity of fake news and political polarization in Brazil, and the lack of research for Portuguese, both dataset and base- line were proposed for Brazilian Portuguese. 1 Introduction Automated fact-checking and news credibility have become undoubtedly an important research issue mainly due to the potential for misinformation to spread in the modern media ecosystem (Guo et al., 2022). Furthermore, although fake news is spread- ing on social media, it is necessary a source me- dia where they would have been posted originally. Since websites have published low-credible news in the past, it is likely to happen again (Baly et al., 2018). Nevertheless, automated news credibility to assist human efforts and increase the understand- ing of the news ecosystem as a whole still requires urgent improvements (Horne et al., 2018). Nowadays, fact-checking organizations have pro- vided lists of unreliable news articles and media 1https://zenodo.org/record/7868597 2https://www.allsides.com/ N. Sentence-level news article Label Title President lowers Brazil’s image with re- peated misinformation and does not receive attention from global leaders. Biased S1 President Jair Bolsonaro touch a sore point of Europeans when he pointed out that the increased use of fossil fuels is a serious en- vironmental setback, in his opening speech at the UN General Assembly, Tuesday (20). Biased S2 Germany received criticism at the UN for the investment agreement with Senegal for the production of gas in the African country. Factual S3 “This constitutes a serious setback for the environment”, he said, referring to the Eu- ropeans Quotes S4 However, Bolsonaro signed measures con- trary to environmental protection during the four years of the Brazilian government. Factual S5 There is a huge difference between speak- ing at the UN and being heard at the UN. Biased Table 1: Sentence-level factuality and bias prediction. sources (Baly et al., 2018). Notwithstanding, these are inefficient once they …

Content truncated to 500 words