Multilingual sentence-level bias detection in Wikipedia

Proceedings of Recent Advances in Natural Language Processing, pages 42–51, Varna, Bulgaria, Sep 2–4, 2019. https://doi.org/10.26615/978-954-452-056-4_006 42 Multilingual Sentence-Level Bias Detection in Wikipedia Desislava Aleksandrova 1,2 Franc¸ois Lareau 1 1 OLST, Universit´e de Montr´eal, first.lastname@umontreal.ca 2 Computer Research Institute of Montreal, first.lastname@crim.ca Pierre-Andr´e M´enard 2 Abstract We propose a multilingual method for the extraction of biased sentences from Wikipedia, and use it to create corpora in Bulgarian, French and English. Sift- ing through the revision history of the ar- ticles that at some point had been con- sidered biased and later corrected, we re- trieve the last tagged and the first un- tagged revisions as the before/after snap- shots of what was deemed a violation of Wikipedia’s neutral point of view policy. We extract the sentences that were re- moved or rewritten in that edit. The ap- proach yields sufficient data even in the case of relatively small Wikipedias, such as the Bulgarian one, where 62k arti- cles produced 5k biased sentences. We evaluate our method by manually anno- tating 520 sentences for Bulgarian and French, and 744 for English. We as- sess the level of noise and analyze its sources. Finally, we exploit the data with well-known classification methods to de- tect biased sentences. Code and datasets are hosted at https://github.com/ crim-ca/wiki-bias. 1 Introduction Our goal is to automatically detect neutral point of view (NPOV) violations at the sentence level with a procedure replicable in multiple languages. Sentence-level bias detection is a type of senti- ment analysis, closely related to subjectivity de- tection (Riloff and Wiebe, 2003; Wiebe and Riloff, 2005; Wilson and Raaijmakers, 2008; Murray and Carenini, 2009; Lin et al., 2011; Al Khatib et al., 2012), where an opinion is considered subjective, and a fact, objective. Yet, as far as bias in writing is concerned, both subjective opinions and objec- tive fact reporting (cf. §5) may, in some cases, be sources of impartiality. The importance of the con- text is one of the main difficulties in detecting bias at the sentence level. Some types of point-of-view bias are equally challenging for humans to detect. Partisanship in editorials, for example, tends to go unnoticed when in line with the reader’s own ideas and beliefs (Yano et al., 2010). A further compli- cation arises from the ambiguity of the term bias, which stands for a lack of fairness or neutrality in realms as varied as human cognition (Tversky and Kahneman, 1974), society (Ross et al., 1977), me- dia (Entman, 2007), internet (Baeza-Yates, 2018; Pitoura et al., 2018) or statistical models and algo- rithms (O’Neil, 2016; Shadowen, 2019), to name a few. With so many different types of bias and their varying definitions, it is not trivial to set the scope of a bias-detection study. The majority of the work on this task is per- formed on news articles (Hirning et al., 2017; Baly et al., 2018; Bellows, 2018) and political blogs (Yano et al., 2010; Iyyer et al., 2014) rather than Wikipedia, because of the relative scarcity of ex- …

Content truncated to 500 words