A Call for Clarity in Contemporary Authorship Attribution Evaluation

Proceedings of Recent Advances in Natural Language Processing, pages 1174–1179 Sep 1–3, 2021. https://doi.org/10.26615/978-954-452-072-4_132 1174 A Call for Clarity in Contemporary Authorship Attribution Evaluation Allen Riddell Indiana University Bloomington Bloomington, Indiana, USA riddella@indiana.edu Haining Wang Indiana University Bloomington Bloomington, Indiana, USA hw56@indiana.edu Patrick Juola EVL Lab, Duquesne University Pittsburgh, Pennsylvania, USA juola@mathcs.duq.edu Abstract Recent research has documented that results reported in frequently-cited authorship attribu- tion papers are difficult to reproduce. Inac- cessible code and data are often proposed as factors which block successful reproductions. Even when original materials are available, problems remain which prevent researchers from comparing the effectiveness of differ- ent methods. To solve the remaining prob- lems—the lack of fixed test sets and the use of inappropriately homogeneous corpora—our paper contributes materials for five closed-set authorship identification experiments. The five experiments feature texts from 106 dis- tinct authors. Experiments involve a range of contemporary non-fiction American English prose. These experiments provide the founda- tion for comparable and reproducible author- ship attribution research involving contempo- rary writing. 1 Introduction Closed-set authorship attribution picks out the likely author of an unsigned document from a pool of candidate authors. Decades of research show that authors leave conspicuous “fingerprints” in their writing (Juola, 2006). A small amount of pre- existing prose (ca. 2,500 words) is often enough to learn enough about the writing “styles” of a set of candidate authors to correctly identify the author of an unsigned document. Authorship at- tribution techniques have found application in nu- merous domains. They have been used to resolve uncertainty about authorship in historical research (Mosteller and Wallace, 1964). For writers liv- ing today, widespread use of authorship attribu- tion techniques—and related author profiling tech- niques (Argamon et al., 2009)—poses a privacy risk (Brennan et al., 2012). A better understand- ing of how authorship attribution techniques work can inform efforts to improve privacy-enhancing language technologies. While there is no doubt that authorship attribu- tion methods have improved over the past century, recent progress is harder to measure. Some of this difficulty is due to the field’s success. In re- cent decades, new methods improve on old ones by small amounts. Given small improvements, as- sessing whether or not the advance may be due to aleatory factors such as preprocessing or a par- ticular dataset becomes difficult. Another reason recent progress is difficult to measure is the lack of standard benchmark tasks. Of 15 frequently-cited authorship attribution studies examined by Potthast et al. (2016), original corpora could be found for only 4 (27%) and code could be located for 0 (0%). While other fields, notably machine translation and language modeling, excel at organizing research ac- tivity around publicly-accessible benchmark tasks, contemporary authorship attribution research has no such tasks. Recent experience suggests that without standard benchmarks—and evidence that researchers can consistently reproduce results using them—a field’s ability to self-assess progress on well-defined tasks can go astray. The field of recommender systems offers a cautionary tale. Rendle et al. (2019) doc- uments a series of papers being published …

Content truncated to 500 words