WikiTalkEdit: A Dataset for modeling Editors’ behaviors on Wikipedia

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2191–2200 June 6–11, 2021. ©2021 Association for Computational Linguistics 2191 WikiTalkEdit: A Dataset for modeling Editors’ behaviors on Wikipedia Kokil Jaidka1 Andrea Ceolin 2 Iknoor Singh3 Niyati Chhaya4 Lyle H. Ungar5 1 Department of Communications and New Media, National University of Singapore 2 Department of Linguistics, University of Pennsylvania 3 Department of Computer Science, University of Sheffield 4 Big Data Experience Lab, Adobe Research India Pvt. Ltd. 5 Department of Computer & Information Science, University of Pennsylvania jaidka@nus.edu.sg Abstract This study introduces and analyzes Wik- iTalkEdit, a dataset of conversations and edit histories from Wikipedia, for research in on- line cooperation and conversation modeling. The dataset comprises dialog triplets from the Wikipedia Talk pages, and editing actions on the corresponding articles being discussed. The exchanges occur between two turn-taking individuals and span all of Wikipedia. We show how the data supports the classic un- derstanding of style matching, where posi- tive emotion and the use of first-person pro- nouns predict a positive emotional change in a Wikipedia contributor. However, they do not predict editorial behavior. On the other hand, feedback invoking evidentiality and crit- icism, and references to Wikipedia’s com- munity norms, is more likely to persuade the contributor to perform edits but is less likely to lead to a positive emotion. We de- veloped baseline classifiers trained on pre- trained RoBERTa features that can predict ed- itorial change with an F1 score of .54, as compared to an F1 score of .66 for predict- ing emotional change. A diagnostic analysis of persisting errors is also provided. We con- clude with possible applications and recom- mendations for future work. The dataset is pub- licly available for the research community at https://github.com/kj2013/WikiTalkEdit/. 1 Introduction Dialogue is a language game of influence, action, and reaction that progresses in a turn-taking man- ner. Persuasion occurs through dialogue when a listener favorably evaluates the authority, claims, and evidentiality through the cues and arguments made by the speaker (Krippendorff, 1993; Schulte, 1980; Durik et al., 2008). Discussions on Wikipedia Talk pages can be use- ful for determining strategies that lead to an im- provement of the article discussed, and for exam- ining if they also lead to an amicable dialogic ex- change. Previous work (Yang et al., 2016a,b, 2017) has explored the role of editors and the types of edits made on Wikipedia, but have not related them to the ongoing conversation on the Wikipedia Talk pages. We introduce the WikiTalkEdit dataset, a novel dataset for research in online collaboration. The dataset is a subset of the Wikipedia Talk Cor- pus available as of May 20181. It contains 12,882 dialogue triples with labels about editors’ sub- sequent editorial (editing) behavior, and 19,632 triplets with labels corresponding to editors’ emo- tion as manifested in their replies. Table 1 has ex- amples from the dataset.2 This new dataset enables various language and behavior modeling tasks. In general, …

Content truncated to 500 words