Annotating and Analyzing Biased Sentences in News Articles using Crowdsourcing

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 1478–1484 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 1478 Annotating and Analyzing Biased Sentences in News Articles using Crowdsourcing Sora Lim1, Adam Jatowt1, Michael F¨arber2, Masatoshi Yoshikawa1 1 Kyoto Univiersity, 2 Karlsruhe Institute of Technology infosora@db.soc.i.kyoto-u.ac.jp, adam@dl.kuis.kyoto-u.ac.jp, michael.faerber@kit.edu, yoshikawa@i.kyoto-u.ac.jp Abstract The spread of biased news and its consumption by the readers has become a considerable issue. Researchers from multiple domains including social science and media studies have made efforts to mitigate this media bias issue. Specifically, various techniques ranging from natural language processing to machine learning have been used to help determine news bias automatically. However, due to the lack of publicly available datasets in this field, especially ones containing labels concerning bias on a fine-grained level (e.g., on sentence level), it is still challenging to develop methods for effectively identifying bias embedded in new articles. In this paper, we propose a novel news bias dataset which facilitates the development and evaluation of approaches for detecting subtle bias in news articles and for understanding the characteristics of biased sentences. Our dataset consists of 966 sentences from 46 English-language news articles covering 4 different events and contains labels concerning bias on the sentence level. For scalability reasons, the labels were obtained based on crowd-sourcing. Our dataset can be used for analyzing news bias, as well as for developing and evalu- ating methods for news bias detection. It can also serve as resource for related researches including ones focusing on fake news detection. Keywords: News Bias, Dataset, Media Bias, Crowd-sourcing 1. Introduction News articles are not always written in a neutral manner, but may deviate from the norm by using dedicated words, a specific writing style, or a preferred author’s viewpoint (Hackett, 1984; Morstatter et al., 2018). Such characteris- tics of media is referred to as media bias and in the con- text of news articles as news bias. News bias has been a challenge for a long time in the world of media. Truthful- ness, fairness, accuracy, and balanced viewpoints have been emphasized in the context of news reporting to avoid news bias, because news can have a large influence on the read- ers, creating viewpoints and attitudes of people towards so- cial issues, and eventually changing political views and the society (Niven, 2002). Recognizing news bias is therefore an important goal in the world of media and it can support (or alert) not only readers consuming news but also help authors to write texts in neutral style. Nowadays, news ar- ticles are mainly published online and are read from var- ious news channels. To monitor and prevent news bias in a timely and efficient manner, first computational ap- proaches (Park et al., 2009; Hutto et al., 2015; Hamborg et al., 2018) have been developed which aim to identify news bias automatically. These methods are based on tech- niques from natural language processing (NLP) and ma- chine learning. However, detecting news bias …

Content truncated to 500 words