WAC: A Corpus of Wikipedia Conversations for Online Abuse Detection

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 1382–1390 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 1382 WAC: A Corpus of Wikipedia Conversations for Online Abuse Detection Noé Cécillon, Vincent Labatut, Richard Dufour and Georges Linarès Laboratoire Informatique d’Avignon – LIA EA 4128, Avignon Université, France {firstname.lastname}@univ-avignon.fr Abstract With the spread of online social networks, it is more and more difficult to monitor all the user-generated content. Automating the moderation process of the inappropriate exchange content on Internet has thus become a priority task. Methods have been proposed for this purpose, but it can be challenging to find a suitable dataset to train and develop them. This issue is especially true for approaches based on information derived from the structure and the dynamic of the conversation. In this work, we propose an original framework, based on the Wikipedia Comment corpus, with comment-level abuse annotations of different types. The major contribution concerns the reconstruction of conversations, by comparison to existing corpora, which focus only on isolated messages (i.e. taken out of their conversational context). This large corpus of more than 380k annotated messages opens perspectives for online abuse detection and especially for context-based approaches. We also propose, in addition to this corpus, a complete benchmarking platform to stimulate and fairly compare scientific works around the problem of content abuse detection, trying to avoid the recurring problem of result replication. Finally, we apply two classification methods to our dataset to demonstrate its potential. Keywords: Wikipedia conversations, Abuse detection, Evaluation framework, Automatic moderation 1. Introduction The ever growing quantity of content posted online requires more and more moderators to monitor this content. A fast and accurate moderation is highly beneficial to online plat- forms, but it is increasingly expensive and difficult to main- tain. Therefore, the automated detection of abusive online content is an important research topic. Corpora allowing to develop such methods often focus on single comments with- out any conversational context (Pavlopoulos et al., 2017; Razavi et al., 2010). Yet, recent works (Papegnies et al., 2019; Yin et al., 2009) suggest that considering the entire conversation thread might improve the automatic detection of abusive content. However, the development of such meth- ods is currently limited by the lack of large scale corpora of conversations. Corpora containing full conversations exists but they have limited number of messages or are not pub- licly available (Napoles et al., 2017; Cécillon et al., 2019). Karan and Šnajder (2019) offers a solution with PreTox, a large corpus of discussion threads from Wikipedia talk pages. However, the quality of their semi-automatically generated annotations might be problematic, as the authors report a Precision of only 51%. In this paper, we reconstruct a large scale corpus of mes- sages from English Wikipedia talk pages, structured as full conversations and annotated with high quality annotations. This results in a corpus containing roughly 193k conversa- tions and 383k messages annotated as being abusive or not. To encourage further development of …

Content truncated to 500 words