Detecting Toxicity in News Articles: Application to Bulgarian

Proceedings of Recent Advances in Natural Language Processing, pages 247–258, Varna, Bulgaria, Sep 2–4, 2019. https://doi.org/10.26615/978-954-452-056-4_029 247 Detecting Toxicity in News Articles: Application to Bulgarian Yoan Dinkov, Ivan Koychev Faculty of Mathematics and Informatics Sofia University Sofia, Bulgaria {jdinkov,koychev}@uni-sofia.bg Preslav Nakov Qatar Computing Research Institute HBKU Doha, Qatar pnakov@hbku.edu.qa Abstract Online media aim for reaching ever big- ger audience and for attracting ever longer attention span. This competition creates an environment that rewards sensational, fake, and toxic news. To help limit their spread and impact, we propose and de- velop a news toxicity detector that can recognize various types of toxic content. While previous research primarily focused on English, here we target Bulgarian. We created a new dataset by crawling a web- site that for five years has been collecting Bulgarian news articles that were manu- ally categorized into eight toxicity groups. Then we trained a multi-class classifier with nine categories: eight toxic and one non-toxic. We experimented with differ- ent representations based on ElMo, BERT, and XLM, as well as with a variety of domain-specific features. Due to the small size of our dataset, we created a sep- arate model for each feature type, and we ultimately combined these models into a meta-classifier. The evaluation results show an accuracy of 59.0% and a macro- F1 score of 39.7%, which represent siz- able improvements over the majority-class baseline (Acc=30.3%, macro-F1=5.2%). 1 Introduction The number of online news sources has grown dra- matically in recent years, and so has the amount of news that has been bombarding users on a daily basis, especially in social media. As people have limited time to spend reading news, capturing peo- ple’s attention is getting ever harder. Media have to use various techniques to get their readers back such as bigger advertisement and better services. Alternatively, it turns out that an easy way to at- tract people’s attention is to use some toxicity in the articles, as people are intrigued by the unusual. Thus, our aim here is to try to detect articles that can harm, give false impressions, or deceive the readers. Such articles can use some of the follow- ing techniques: • Sensationalism: overexposing insignificant or ordinary events by manipulating the main point of an article; • Fake news: news that sound right, but totally misinterpret facts such as statistical data, lo- cations, or dates, with the conscious aim of proving something wrong. • Conspiracy theories: information that usu- ally gives a lot of detail, but does not offer officially confirmed evidence or scientific re- search to back the claims that are being made. This is typically centered around political, or strange scientific phenomena. • Hate speech: specifically targeting a person or a social group to brutalize them or to bully over the rate of normal conversation, to di- rectly hurt or to manipulate them. Given the proliferation of toxic news online, there have been many efforts to create tools and mechanisms to counteract their effect and spread. Such tools should help preserve …

Content truncated to 500 words