What Can We Learn from Noun Substitutions in Revision Histories?

Proceedings of the 28th International Conference on Computational Linguistics, pages 1359–1370 Barcelona, Spain (Online), December 8-13, 2020 1359 What Can We Learn from Noun Substitutions in Revision Histories? Talita Rani Anthonio Michael Roth Institute for Natural Language Processing University of Stuttgart {anthonta,rothml}@ims.uni-stuttgart.de Abstract In community-edited resources such as wikiHow, sentences are subject to revisions on a daily basis. Recent work has shown that resulting improvements over time can be modelled compu- tationally, assuming that each revision contributes to the improvement. We take a closer look at a subset of such revisions, for which we attempt to improve a computational model and validate in how far the assumption that ‘revised means better’ actually holds. The subset of revisions considered here are noun substitutions, which often involve interesting semantic relations, in- cluding synonymy, antonymy and hypernymy. Despite the high semantic relatedness, we find that a supervised classifier can distinguish the revised version of a sentence from an original ver- sion with an accuracy close to 70%, when taking context into account. In a human annotation study, we observe that annotators identify the revised sentence as the ‘better version’ with similar performance. Our analysis reveals a fair agreement among annotators when a revision improves fluency. In contrast, noun substitutions that involve common lexical-semantic relationships are often perceived as being equally good or tend to cause disagreements. While these findings are also reflected in classification scores, a comparison of results shows that our model fails in cases where humans can resort to factual knowledge or intuitions about the required level of specificity. 1 Introduction Instructional texts often go through multiple revisions. For their final form, the author has to ensure that it is clear in which sequence described actions need to be performed and in what manner they have to be executed. A collection of revisions of instructional texts can be found, for instance, on the community-edited platform wikiHow. In previous work using wikiHow revision histories, we showed that improvements in texts can be modeled computationally under the assumption that revised sentences are better than their predecessors (Anthonio et al., 2020). This assumption seems intuitive for edits that correct or clarify parts of a sentence (e.g., in terms of grammar/factual mistakes). In the analysis, however, we also found a number of edits that involve paraphrases or more subtle differences in terms of specificity/genericity. In these cases, it remains unclear in how far edits actually represent improvements. This problem notably exists also in other work on textual revisions which aim to develop a compu- tational model, including studies on Wikipedia (Bronner and Monz, 2012; Daxenberger and Gurevych, 2012; Faruqui et al., 2018) and ArgRewrite (Zhang and Litman, 2015). In particular, these studies ne- glect two important aspects: First, they merely categorize edits by how they change a sentence (e.g., information deletion/addition, link creation/deletion), without specifying if or in how far the change rep- resents an actual improvement. Secondly, they provide limited or no insight about how the content of a revised sentence is semantically related …

Content truncated to 500 words