One Document, Many Revisions: A Dataset for Classification and Description of Edit Intents

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 5517–5524 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 5517 One Document, Many Revisions: A Dataset for Classification and Description of Edit Intents Dheeraj Rajagopal∗, Xuchao Zhang♣, Michael Gamon†, Sujay Kumar Jauhar†, Diyi Yang♠, Eduard Hovy∗ ∗Carnegie Mellon University ♠Georgia Institute of Technology†Microsoft Research ♣NEC Labs America {dheeraj, hovy}@cs.cmu.edu xuczhang@nec-labs.com {mgamon,sjauhar}@microsoft.com diyi.yang@cc.gatech.edu Abstract Document authoring involves a lengthy revision process, marked by individual edits that are frequently linked to comments. Modeling the relationship between edits and comments leads to a better understanding of document evolution, potentially benefiting applications such as content summarization, and task triaging. Prior work on understanding revisions has primarily focused on classifying edit intents, but falling short of a deeper understanding of the nature of these edits. In this paper, we present explore the challenge of describing an edit at two levels: identifying the edit intent, and describing the edit using free-form text. We begin by defining a taxonomy of general edit intents and introduce a new dataset of full revision histories of Wikipedia pages, annotated with each revision’s edit intent. Using this dataset, we train a classifier that achieves a 90% accuracy in identifying edit intent. We use this classifier to train a distantly-supervised model that generates a high-level description of a revision in free-form text. Our experimental results show that incorporating edit intent information aids in generating better edit descriptions. We establish a set of baselines for the edit description task, achieving a best score of 28 ROUGE, thus demonstrating the effectiveness of our layered approach to edit understanding. Keywords: Edit Description, Wiki Edits, Comment Generation 1. Introduction A typical written document undergoes multiple revi- sions before reaching a final version. When multiple authors do these revisions, it can be difficult to track them and understand how the document evolved. A method to automatically identify the edit intent and the action would be of use. Further, such a method would support an eventual system that could automatically de- cide which sentences to edit and how to edit them. Revisions have been shown to provide insights into un- derstanding the relationship between authors and doc- ument evolution Yang et al. [2016], article trustwor- thiness Zeng et al. [2006], neutrality or bias point-of- view detection Recasens et al. [2013], and author par- ticipation Halfaker et al. [2013]. Wikipedia is a large- scale community-maintained resource that uses revi- sions and comments to build a successful encyclope- dic resource. When an author edits a page for an edit or revision, she/he usually leaves a comment, which is free-form text description of the edit that typically characterizes it’s intent. A typical edit example from Wikipedia is shown in figure 1. Prior work on revision- intents primarily develop classification systems based on edit intents Faigley and Witte [1981], Yang et al. [2016], Daxenberger and Gurevych [2012]. However, capturing the intent alone does not adequately charac- terize the nature of the edit. In this …

Content truncated to 500 words