Authorship Attribution for Neural Text Generation

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 8384–8395, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 8384 Authorship Attribution for Neural Text Generation Adaku Uchendu Thai Le Kai Shu† Dongwon Lee The Pennsylvania State University, University Park, PA, USA {azu5030, thaile, dongwon}@psu.edu Illinois Institute of Technology, Chicago, IL, USA† kshu@iit.edu† Abstract In recent years, the task of generating realis- tic short and long texts have made tremendous advancements. In particular, several recently proposed neural network-based language mod- els have demonstrated their astonishing capa- bilities to generate texts that are challenging to distinguish from human-written texts with the naked eye. Despite many benefits and util- ities of such neural methods, in some applica- tions, being able to tell the “author” of a text in question becomes critically important. In this work, in the context of this Turing Test, we investigate the so-called authorship attri- bution problem in three versions: (1) given two texts T1 and T2, are both generated by the same method or not? (2) is the given text T written by a human or machine? (3) given a text T and k candidate neural methods, can we single out the method (among k alterna- tives) that generated T? Against one human- written and eight machine-generated texts (i.e., CTRL, GPT, GPT2, GROVER, XLM, XL- NET, PPLM, FAIR), we empirically experi- ment with the performance of various mod- els in three problems. By and large, we find that most generators still generate texts sig- nificantly different from human-written ones, thereby making three problems easier to solve. However, the qualities of texts generated by GPT2, GROVER, and FAIR are better, often confusing machine classifiers in solving three problems. All codes and datasets of our exper- iments are available at: https://bit.ly/ 302zWdz 1 Introduction Recent rapid advancements in deep learning tech- nologies have enabled the generation of realistic artifacts (e.g., Deepfakes) that are difficult to dis- tinguish from genuine human-generated artifacts. In the text domain, which is the main focus of this work, similarly, the advancement of Natural Lan- guage Generation (NLG), especially those based on neural language models, has led to the inunda- tion of realistic text generation. As novel NLG techniques become more sophis- ticated and prevalent, corresponding pitfalls and risks of such technologies also increase. Adver- saries may use such technologies to generate realis- tic artifacts to trick naive users in fraudulent activi- ties (e.g., machine-generated chatbot conversation in a phishing scam or deepfake-based disinforma- tion campaign). Therefore, the need to distinguish machine-generated texts from human-written ones, so-called the Turing Test, naturally arises. Further- more, in some security applications, merely being able to identify machine-generated text may not be sufficient. Instead, a more critical solution would be to tell which NLG method among many can- didates has generated a given text in question–so- called the Authorship Attribution (AA) problem. To improve our understanding of this newly-emerging problem, we empirically investigate three versions of the AA problem in this paper. For all three ver- sions, …

Content truncated to 500 words