User Edits Classification Using Document Revision Histories

Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics, pages 356–366, Avignon, France, April 23 - 27 2012. c⃝2012 Association for Computational Linguistics User Edits Classification Using Document Revision Histories Amit Bronner Informatics Institute University of Amsterdam a.bronner@uva.nl Christof Monz Informatics Institute University of Amsterdam c.monz@uva.nl Abstract Document revision histories are a useful and abundant source of data for natural language processing, but selecting relevant data for the task at hand is not trivial. In this paper we introduce a scalable ap- proach for automatically distinguishing be- tween factual and fluency edits in document revision histories. The approach is based on supervised machine learning using lan- guage model probabilities, string similar- ity measured over different representations of user edits, comparison of part-of-speech tags and named entities, and a set of adap- tive features extracted from large amounts of unlabeled user edits. Applied to con- tiguous edit segments, our method achieves statistically significant improvements over a simple yet effective edit-distance base- line. It reaches high classification accuracy (88%) and is shown to generalize to addi- tional sets of unseen data. 1 Introduction Many online collaborative editing projects such as Wikipedia1 keep track of complete revision histo- ries. These contain valuable information about the evolution of documents in terms of content as well as language, style and form. Such data is publicly available in large volumes and constantly grow- ing. According to Wikipedia statistics, in August 2011 the English Wikipedia contained 3.8 million articles with an average of 78.3 revisions per ar- ticle. The average number of revision edits per month is about 4 million in English and almost 11 million in total for all languages.2 1http://www.wikipedia.org 2Average for the 5 years period between August 2006 and August 2011. The count includes edits by registered Exploiting document revision histories has proven useful for a variety of natural language processing (NLP) tasks, including sentence com- pression (Nelken and Yamangil, 2008; Yamangil and Nelken, 2008) and simplification (Yatskar et al., 2010; Woodsend and Lapata, 2011), informa- tion retrieval (Aji et al., 2010; Nunes et al., 2011), textual entailment recognition (Zanzotto and Pen- nacchiotti, 2010), and paraphrase extraction (Max and Wisniewski, 2010; Dutrey et al., 2011). The ability to distinguish between factual changes or edits, which alter the meaning, and flu- ency edits, which improve the style or readability, is a crucial requirement for approaches exploit- ing revision histories. The need for an automated classification method has been identified (Nelken and Yamangil, 2008; Max and Wisniewski, 2010), but to the best of our knowledge has not been di- rectly addressed. Previous approaches have either applied simple heuristics (Yatskar et al., 2010; Woodsend and Lapata, 2011) or manual annota- tions (Dutrey et al., 2011) to restrict the data to the type of edits relevant to the NLP task at hand. The work described in this paper shows that it is possible to automatically distinguish between fac- tual and fluency edits. This is very desirable as it does not rely on …

Content truncated to 500 words