Modeling the Relationship between User Comments and Edits in Document Revision

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 5002–5011, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 5002 Modeling the Relationship between User Comments and Edits in Document Revision Xuchao Zhang†∗, Dheeraj Rajagopal‡∗, Michael Gamon§, Sujay Kumar Jauhar§ and Chang-Tien Lu† †Discovery Analytics Center, Virginia Tech, Falls Church, VA, USA ‡Carnegie Mellon University, Pittsburgh, PA, USA §Microsoft Research, Redmond, WA, USA †{xuczhang, ctlu}@vt.edu, ‡dheeraj@cs.cmu.edu, §{mgamon, sjauhar}@microsoft.com Abstract Management of collaborative documents can be difficult, given the profusion of edits and comments that multiple authors make during a document’s evolution. Reliably modeling the relationship between edits and comments is a crucial step towards helping the user keep track of a document in flux. A number of au- thoring tasks, such as categorizing and sum- marizing edits, detecting completed to-dos, and visually rearranging comments could ben- efit from such a contribution. Thus, in this pa- per we explore the relationship between com- ments and edits by defining two novel, related tasks: Comment Ranking and Edit Anchoring. We begin by collecting a dataset with more than half a million comment-edit pairs based on Wikipedia revision histories. We then pro- pose a hierarchical multi-layer deep neural- network to model the relationship between ed- its and comments. Our architecture tackles both Comment Ranking and Edit Anchoring tasks by encoding specific edit actions such as additions and deletions, while also accounting for document context. In a number of evalu- ation settings, our experimental results show that our approach outperforms several strong baselines significantly. We are able to achieve a precision@1 of 71.0% and a precision@3 of 94.4% for Comment Ranking, while we achieve 74.4% accuracy on Edit Anchoring. 1 Introduction Comments are widely used in collaborative docu- ment writing as a natural way to suggest and track changes, annotate content and explain the intent of edits. Table 1 shows an example of a comment left by an editor in a Wikipedia revision. The com- ment explains the intent of adding a missing re- searcher’s name (Sutskever) to a citation. ∗ The work was done while these authors interned in Microsoft Research. This example demonstrates that user comments closely relate to the intent of an edit, the actual edit operation, and the content and location in the document that underwent change. However, dur- ing collaborative document authoring, the tracking and maintenance of comments becomes increas- ingly more challenging due to the large number of edits and comments that authors make. For example, structurally refactoring a document can significantly change the order of paragraphs and sentences, stranding comments in confusing and contextually inappropriate locations. Or, com- ments may have already been addressed but con- tinue to linger in the document without having been marked as completed. These issues, among others, are exacerbated when multiple authors col- laborate on a document, often asynchronously. It becomes difficult to know which tasks have been completed and which haven’t, especially if authors are not …

Content truncated to 500 words