Towards Modeling Revision Requirements in wikiHow Instructions

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 8407–8414, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 8407 Towards Modeling Revision Requirements in wikiHow Instructions Irshad Ahmad Bhat Talita Rani Anthonio University of Stuttgart Institute for Natural Language Processing {bhatid,anthonta,rothml}@ims.uni-stuttgart.de Michael Roth Abstract wikiHow is a resource of how-to guides that describe the steps necessary to accomplish a goal. Guides in this resource are regularly edited by a community of users, who try to improve instructions in terms of style, clarity and correctness. In this work, we test whether the need for such edits can be predicted auto- matically. For this task, we extend an existing resource of textual edits with a complementary set of approx. 4 million sentences that remain unedited over time and report on the outcome of two revision modeling experiments. 1 Introduction Instructional texts have become an integral part of our daily lives, be it in the form of assembly in- structions, product leaflets, troubleshooting guides, or board game manuals. A key property across all types of such texts is that they must be clear enough so that readers can actually achieve the goal described by the instructions. Previous studies in computational linguistics have dealt with the clarity of specific types of in- structions, such as route directions (Byron et al., 2009; Striegnitz et al., 2011) and software require- ments (Willis et al., 2008; Yang et al., 2010). As an indicator for clarity, they relied either on successful execution in a virtual environment or on manual annotations of predefined ambiguity types. A large and more general dataset of instructional texts, wik- iHowToImprove, has recently been introduced by Anthonio et al. (2020). wikiHowToImprove con- sists of edits for about 2.5 million sentences derived automatically from revision histories of wikiHow, a collaboratively edited platform of how-to guides. In a set of human and computational experiments, Anthonio et al. (2020) show that such edits are of- ten made to clarify or correct a sentence and that the difference between an “older” and “newer” ver- sion of a sentence can be predicted computationally. We address two notable questions, using the work of Anthonio et al. (2020) as a starting point: (1) Are results for the task of distinguishing two versions of a sentence (henceforth version distinc- tion) specific to instructional texts, such as their guides from wikiHow, or can underlying linguistic characteristics be modelled to a similar extent in a different text genre? We reproduce version dis- tinction results for different computational models on a variant of wikiHowToImprove and provide comparison results on an earlier dataset of revision edits (WikiAtomicEdits) derived from Wikipedia (Faruqui et al., 2018). In our experiments, we find that models for version distinction work best on instructional texts and that they are capable of de- tecting a variety of potential reasons for revision, including grammatical errors, semantic implausi- bilities and vague expressions. (2) Given the results on instructional texts, is it possible to model whether a sentence requires re- vision in …

Content truncated to 500 words