StRE: Self Attentive Edit Quality Prediction in Wikipedia

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3962–3972 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 3962 StRE: Self Attentive Edit Quality Prediction in Wikipedia Soumya Sarkar∗1, Bhanu Prakash Reddy*2, Sandipan Sikdar3 Animesh Mukherjee4 IIT Kharagpur, India1,2,4,RWTH Aachen, Germany3 soumya015@iitkgp.ac.in1,bhanu77prakash@gmail.com2 sandipan.sikdar@cssh.rwth-aachen.de3, animesh@cse.iitkgp.ac.in4 Abstract Wikipedia can easily be justified as a behe- moth, considering the sheer volume of con- tent that is added or removed every minute to its several projects. This creates an immense scope, in the field of natural language pro- cessing toward developing automated tools for content moderation and review. In this paper we propose Self Attentive Revision Encoder (StRE) which leverages orthographic similar- ity of lexical units toward predicting the qual- ity of new edits. In contrast to existing propo- sitions which primarily employ features like page reputation, editor activity or rule based heuristics, we utilize the textual content of the edits which, we believe contains superior signatures of their quality. More specifically, we deploy deep encoders to generate repre- sentations of the edits from its text content, which we then leverage to infer quality. We further contribute a novel dataset containing ∼ 21M revisions across 32K Wikipedia pages and demonstrate that StRE outperforms exist- ing methods by a significant margin – at least 17% and at most 103%. Our pre-trained model achieves such result after retraining on a set as small as 20% of the edits in a wikipage. This, to the best of our knowledge, is also the first at- tempt towards employing deep language mod- els to the enormous domain of automated con- tent moderation and review in Wikipedia. 1 Introduction Wikipedia is the largest multilingual encyclopedia known to mankind with the current English ver- sion consisting of more than 5M articles on highly diverse topics which are segregated into cate- gories, constructed by a large editor base of more than 32M editors (Hube and Fetahu, 2019). To en- courage transparency and openness, Wikipedia al- lows anyone to edit its pages albeit with certain ∗*Both authors contributed equally guidelines for them1. Problem: The inherent openness of Wikipedia has also made it vulnerable to external agents who intentionally attempt to divert the unbiased, ob- jective discourse to a narrative which is aligned with the interest of the malicious actors. Our pilot study on manually annotated 100 Wikipedia pages of four categories (25 pages each category) shows us that at most 30% of the edits are reverted (See Fig 1). Global average of number of reverted dam- aging edits is ∼ 9%2. This makes manual interven- tion to detect these edits with potential inconsis- tent content, infeasible. Wikipedia hence deploys machine learning based classifiers (West et al., 2010; Halfaker and Taraborelli, 2015) which pri- marily leverage hand-crafted features from three aspects of revision (i) basic text features like re- peated characters, long words, capitalized words etc. (ii) temporal features like inter arrival time be- tween events of interest (iii) dictionary based fea- tures like presence …

Content truncated to 500 words