Generalizing Unmasking for Short Texts

Proceedings of NAACL-HLT 2019, pages 654–659 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 654 Generalizing Unmasking for Short Texts Janek Bevendorff ∗ Benno Stein∗ Matthias Hagen† Martin Potthast‡ ∗Bauhaus-Universit¨at Weimar †Martin-Luther-Universit¨at Halle-Wittenberg ‡Leipzig University <first>.<last>@uni-{weimar, leipzig}.de <first>.<last>@informatik.uni-halle.de Abstract Authorship verification is the problem of in- ferring whether two texts were written by the same author. For this task, unmasking is one of the most robust approaches as of to- day with the major shortcoming of only be- ing applicable to book-length texts. In this pa- per, we present a generalized unmasking ap- proach which allows for authorship verifica- tion of texts as short as four printed pages with very high precision at an adjustable re- call tradeoff. Our generalized approach there- fore reduces the required material by orders of magnitude, making unmasking applicable to authorship cases of more practical proportions. The new approach is on par with other state-of- the-art techniques that are optimized for texts of this length: it achieves accuracies of 75– 80 %, while also allowing for easy adjustment to forensic scenarios that require higher levels of confidence in the classification. 1 Introduction With advances in computational stylometry, deter- mining the original authorship of unknown literary publications can be accomplished with near cer- tainty by state-of-the-art authorship verification. If the source material is abundant and sufficiently many known publications exist, linking them to- gether is hardly a challenge. But the playing field changes entirely if only fragments are available for verification, either because few known works of an author exist or because the text to be veri- fied is only a few pages long. With classification results significantly above chance, yet far below certainty, verification approaches struggle to pro- duce reliable results in short-text scenarios and thus lack real-world practicality for material far below book length. Even the unmasking approach by Koppel and Schler (2004), which otherwise proved to be one of the most ingenious and robust veri- fication approaches, fails in this setting and can only deliver below-average performance compared to more specialized verification systems, which still display high uncertainty themselves with an error of up to 25 % (Stamatatos et al., 2015). At PAN 2015, individual texts of the English-language dataset had an average size of about 1.5 kB (less than 400 words), so it does not come as a surprise that none of the participants employed unmasking. To tackle the uncertainty problem of authorship verification on short texts, we propose a general- ized unmasking approach which prioritizes pre- cision so as to verify authorship with reliable re- sults while rejecting cases of low certainty. We also present a new open-source general-purpose unmasking framework as a highly-customizable implementation of our approach.1 2 Related Work Authorship analysis has been practiced since the late 19th century (Bourne, 1897). Although mostly the narrower task of authorship attribution has been considered, where texts are attributed to a set of given authors, recently, authorship verifica- tion has been proposed as a more fundamental task. …

Content truncated to 500 words