wikiHowToImprove: A Resource and Analyses on Edits in Instructional Texts

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 5721–5729 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 5721 wikiHowToImprove: A Resource and Analyses on Edits in Instructional Texts Talita Rani Anthonio∗, Irshad Ahmad Bhat∗, Michael Roth University of Stuttgart Institute for Natural Language Processing {anthonta,bhatid,rothml}@ims.uni-stuttgart.de Abstract Instructional texts, such as articles in wikiHow, describe the actions necessary to accomplish a certain goal. In wikiHow and other resources, such instructions are subject to revision edits on a regular basis. Do these edits improve instructions only in terms of style and correctness, or do they provide clarifications necessary to follow the instructions and to accomplish the goal? We describe a resource and first studies towards answering this question. Specifically, we create wikiHowToImprove, a collection of revision histories for about 2.7 million sentences from about 246 000 wikiHow articles. We describe human annotation studies on categorizing a subset of sentence-level edits and provide baseline models for the task of automatically distinguishing “older” from “newer” versions of a sentence. Keywords: Corpus creation, Semantics, Other 1. Introduction The ever increasing size of the World Wide Web has made it possible to find instructional texts, or how-to guides, on practically any topic or activity. wikiHow is an online plat- form on which a community of users collaboratively write such guides. As of March 2020, wikiHow consists of more than 246 000 articles.1 Factors that constitute good instruc- tional texts have been studied across various disciplines for decades, including inference requirements in cognitive sci- ence (Britton et al., 1990); document design in educational research (Misanchuk, 1992); and motivational processes in sociology (Guthrie et al., 2004). Yet, it remains open what linguistic phenomena are involved in these factors and whether they can be detected and handled automatically. A first step towards filling this gap is to compare changes made across multiple versions of the same set of instruc- tions, under the assumption that later versions are improve- ments over a first version.2 Recent work on revisions in Wikipedia has shown that changes indeed serve a clarify- ing function (Faruqui et al., 2018). According to that study, however, most changes in Wikipedia provide new informa- tion (43%), with refinements only ranking second (24%). The function and information provided by revisions is po- tentially different in context of how-to guides, as their content is largely independent of factual knowledge that changes over time. Therefore, we perform a study simi- lar to that by Faruqui et al. (2018) on wikiHow articles. As our goal is to find observable patterns that reflect potential improvements, we further attempt to sub-categorize edits according to the information changed between two versions of a wikiHow article. For this step, we create a dataset of sentence-level revisions for each article in wikiHow. An ex- ample is shown in Table 1. On this dataset, we carry out two types of studies: first, we perform annotation experiments to find out more about the types of changes and proportions ∗ …

Content truncated to 500 words