A Multifaceted Framework to Evaluate Evasion, Content Preservation, and Misattribution in Authorship Obfuscation Techniques

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 2391 - 2406 December 7-11, 2022 ©2022 Association for Computational Linguistics A Multifaceted Framework to Evaluate Evasion, Content Preservation, and Misattribution in Authorship Obfuscation Techniques Malik H. Altakrori School of Computer Science McGill University / Mila Montreal, Canada malik.altakrori@mail.mcgill.ca Thomas Scialom Meta AI Paris, France tscialom@meta.com Benjamin C. M. Fung School of Information Studies McGill University / Mila Montreal, Canada ben.fung@mcgill.ca Jackie Chi Kit Cheung School of Computer Science McGill University / Mila Montreal, Canada jcheung@cs.mcgill.ca Abstract Authorship obfuscation techniques have com- monly been evaluated based on their ability to hide the author’s identity (evasion) while pre- serving the content of the original text. How- ever, to avoid overstating the systems’ effec- tiveness, evasion detection must be evaluated using competitive identification techniques in settings that mimic real-life scenarios, and the outcomes of the content-preservation evalua- tion have to be interpretable by potential users of these obfuscation tools. Motivated by re- cent work on cross-topic authorship identifi- cation and content preservation in summariza- tion, we re-evaluate different authorship ob- fuscation techniques on detection evasion and content preservation. Furthermore, we pro- pose a new information-theoretic measure to characterize the misattribution harm that can be caused by detection evasion. Our results1 reveal key weaknesses in state-of-the-art ob- fuscation techniques and a surprisingly com- petitive effectiveness from a back-translation baseline in all evaluation aspects. 1 Introduction Authorship obfuscation is the task of masking the writing style of an author of a document to pre- vent authorship identification techniques from us- ing stylistic patterns to reveal the author’s iden- tity (Kacmarcik and Gamon, 2006). The motivation for this task is to protect the public from the misuse of authorship identification techniques to suppress freedom of speech or to persecute whistle-blowers. When a new authorship obfuscation technique is proposed, it is crucial to compare its effectiveness to state-of-the-art obfuscation tools in settings that 1The code is available on https://malikaltakrori. github.io/papers/EMNLP2022/. accurately depict the real-life environment where such a tool may be used. One important assump- tion that should be made is that a de-anonymizer is likely to use the most competitive authorship iden- tification tool available to identify the author of a document. Using an inferior or brittle authorship at- tribution technique, or a weak obfuscation baseline will overstate the performance of such obfuscation tools. For example, previous work on obfuscation has evaluated obfuscation techniques against an identification tool that uses the exact same features and classifier that were used to obfuscate it in the first place (Mahmood et al., 2019). This could lead to misleadingly high impression of the system’s effectiveness. Similarly, obfuscation techniques must convey the same intended message both before and after obfuscation. Therefore, when a new obfuscation tool is proposed, it is evaluated on both the qual- ity of obfuscation and its ability to preserve the content. With the recent development in language models and their ability to generate text, many au- tomatic measures have been introduced to evaluate …

Content truncated to 500 words