Rethinking the Authorship Verification Experimental Setups

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 5634 - 5643 December 7-11, 2022 ©2022 Association for Computational Linguistics Rethinking the Authorship Verification Experimental Setups Florin Brad∗ Bitdefender Andrei Manolache∗ Bitdefender, University of Stuttgart {fbrad,amanolache,eburceanu}@bitdefender.com Elena Burceanu Bitdefender Antonio Barbalau University of Bucharest abarbalau@fmi.unibuc.ro Radu Tudor Ionescu University of Bucharest raducu.ionescu@gmail.com Marius Popescu University of Bucharest popescunmarius@gmail.com Abstract One of the main drivers of the recent advances in authorship verification is the PAN large-scale authorship dataset. Despite generating signif- icant progress in the field, inconsistent perfor- mance differences between the closed and open test sets have been reported. To this end, we improve the experimental setup by proposing five new public splits over the PAN dataset, specifically designed to isolate and identify bi- ases related to the text topic and to the author’s writing style. We evaluate several BERT-like baselines on these splits, showing that such models are competitive with authorship verifi- cation state-of-the-art methods. Furthermore, using explainable AI, we find that these base- lines are biased towards named entities. We show that models trained without the named entities obtain better results and generalize bet- ter when tested on DarkReddit, our new dataset for authorship verification. 1 Introduction Identifying the author of a text is one of the most versatile NLP tasks, with applications ranging from plagiarism detection to forensics and monitoring the activity of cyber-criminals. The task spans sev- eral decades and was tackled using statistical lin- guistics (Mendenhall, 1887; Zipf, 1932; Mosteller and Wallace, 1964), and, more recently, machine learning (de Vel et al., 2001; Zhao and Zobel, 2005; Koppel et al., 2007; Stamatatos, 2009). Due to the typically small data setup of authorship analysis tasks, deep learning methods had a slow start in this domain. Nevertheless, inspired by the impres- sive performance of pre-trained language models, such as BERT (Devlin et al., 2019), these meth- ods gained traction in authorship analysis as well. Saedi and Dras (2021) showed that Convolutional Siamese Networks are more robust than a BERT- based method over large-scale authorship attribu- tion tasks. ∗Equal contribution. Barlas and Stamatatos (2020) investigated pre- trained language models for cross-topic and cross- domain authorship attribution and showed that BERT and ELMo (Peters et al., 2018) achieve the best results while being the most stable approaches. Fabien et al. (2020) introduced BERT for Author- ship Attribution (BertAA) in which they combine BERT with stylometric features for authorship at- tribution. The authors remarked that their model is unable to perform text similarity evaluation in the context of the more difficult authorship verification problem, which we tackle. One of the main contributors to the active devel- opments in authorship analysis is the PAN organiz- ing team, who proposed annual shared tasks since 2009. While the recent PAN 2020 and 2021 con- tests increased the difficulty of the authorship ver- ification task and enabled large-scale model train- ing (Kestemont et al., 2020, 2021), there are still possible generalization issues due to the dataset splits. For instance, models …

Content truncated to 500 words