The Impact of Topic Bias on Quality Flaw Prediction in Wikipedia

Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics, pages 721–730, Sofia, Bulgaria, August 4-9 2013. c⃝2013 Association for Computational Linguistics The Impact of Topic Bias on Quality Flaw Prediction in Wikipedia Oliver Ferschke†, Iryna Gurevych†‡ and Marc Rittberger‡ † Ubiquitous Knowledge Processing Lab Department of Computer Science, Technische Universit¨at Darmstadt ‡ Information Center for Education German Institute for Educational Research and Educational Information http://www.ukp.tu-darmstadt.de Abstract With the increasing amount of user gener- ated reference texts in the web, automatic quality assessment has become a key chal- lenge. However, only a small amount of annotated data is available for training quality assessment systems. Wikipedia contains a large amount of texts anno- tated with cleanup templates which iden- tify quality flaws. We show that the dis- tribution of these labels is topically bi- ased, since they cannot be applied freely to any arbitrary article. We argue that it is necessary to consider the topical restric- tions of each label in order to avoid a sam- pling bias that results in a skewed classifier and overly optimistic evaluation results. We factor out the topic bias by extracting reliable training instances from the revi- sion history which have a topic distribu- tion similar to the labeled articles. This ap- proach better reflects the situation a classi- fier would face in a real-life application. 1 Introduction User generated content is the main driving force of the increasingly social web. Blogs, wikis and forums make up a large amount of the daily infor- mation consumed by web users. The main proper- ties of user generated content are a low publication threshold and little or no editorial control, which leads to a high variance in quality. In order to nav- igate through large repositories of information effi- ciently and safely, users need a way to quickly as- sess the quality of the content. Automatic quality assessment has therefore become a key application in today’s information society. However, there is a lack of training data annotated with fine-grained quality information. Wikipedia, the largest encyclopedia on the web, contains so-called cleanup templates, which con- stitute a sophisticated system of user generated la- bels that mark quality problems in articles. Re- cently, these cleanup templates have been used for automatically identifying articles with particular quality flaws in order to support Wikipedia’s qual- ity assurance process in Wikipedia. In a shared task (Anderka and Stein, 2012b), several systems have shown that it is possible to identify the ten most frequent quality flaws with high recall and fair precision. However, quality flaw detection based on cleanup template recognition suffers from a topic bias that is well known from other text classifica- tion applications such as authorship attribution or genre identification. We discovered that cleanup templates have implicit topical restrictions, i.e. they cannot be applied to any arbitrary article. As a consequence, corpora of flawed articles based on these templates are biased towards particular topics. We argue that it is therefore not sufficient for evaluating a quality flaw prediction …

Content truncated to 500 words