Bias Analysis and Mitigation in the Evaluation of Authorship Verification

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6301–6306 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 6301 Bias Analysis and Mitigation in the Evaluation of Authorship Verification Janek Bevendorff ∗ Benno Stein∗ Matthias Hagen† Martin Potthast‡ ∗Bauhaus-Universität Weimar †Martin-Luther-Universität Halle-Wittenberg ‡Leipzig University <first>.<last>@uni-{weimar, leipzig}.de <first>.<last>@informatik.uni-halle.de Abstract The PAN series of shared tasks is well known for its continuous and high quality research in the field of digital text forensics. Among others, PAN contributions include original cor- pora, tailored benchmarks, and standardized experimentation platforms. In this paper we review, theoretically and practically, the au- thorship verification task and conclude that the underlying experiment design cannot guaran- tee pushing forward the state of the art—in fact, it allows for top benchmarking with a sur- prisingly straightforward approach. In this re- gard, we present a “Basic and Fairly Flawed” (BAFF) authorship verifier that is on a par with the best approaches submitted so far, and that illustrates sources of bias that should be elim- inated. We pinpoint these sources in the eval- uation chain and present a refined authorship corpus as effective countermeasure. 1 Introduction When tackling a problem in empirical research, a sound and reliable evaluation of competing solution approaches is a prerequisite to achieve agreement on the state-of-the-art performance. For authorship verification, the PAN series of shared tasks caters for the most important benchmarks to which new approaches refer and compare against. The fun- damental problem in authorship verification is to decide whether two given texts were written by the same author. When experimenting within the PAN setting, we learned that one can quickly achieve a competitive performance for this task—with one of the most basic approaches: a TFIDF-weighted character 3-gram model. By extending this model with a few additional features, such as the Kullback- Leibler divergence and related measures, we were able to reach the performance of the best verifiers submitted so far.1 However, reality caught up with us when we applied our verifier to other author- ship verification problems with little success. To 1https://www.tira.io/task/authorship-verification/ get to the bottom of this rather baffling outcome, we carried out a systematic analysis of the entire evaluation chain, its problem definition, its cor- pora, its evaluation procedure, and of course our model, in search of any sources of bias that may have artificially inflated the performance of our approach. The paper in hand introduces our “Ba- sic and Fairly Flawed” (BAFF) model and reports on our bias analysis. Moreover, in an attempt to improve the situation and call for better data, we not only contribute a new and carefully curated authorship verification corpus,2 but also collect a few best practices for the creation of such corpora. The outlined situation calls into question a lot of what we believed to know about the state of the art, and future PAN tasks on verification will have to rectify these issues in order to provide for a more valid assessment of the state of the …

Content truncated to 500 words