CCTAA: A Reproducible Corpus for Chinese Authorship Attribution Research

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 5889–5893 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 5889 CCTAA: A Reproducible Corpus for Chinese Authorship Attribution Research Haining Wang, Allen Riddell Indiana University Bloomington Bloomington, IN, USA {hw56|riddella}@indiana.edu Abstract Authorship attribution infers the likely author of an unsigned, single-authored document from a pool of candidates. Despite recent advances, a lack of standard, reproducible testbeds for Chinese language documents impedes progress. In this paper, we present the Chinese Cross-Topic Authorship Attribution (CCTAA) corpus. It is the first standard testbed for authorship attribution on contemporary Chinese prose. The cross-topic design and relatively inflexible genre of newswire contribute to an appropriate level of difficulty. It supports reproducible research by using pre-defined data splits. We show that a sequence classifier based on pre-trained Chinese RoBERTa embedding and a support vector machine classifier using function character n-gram frequency features perform below expectations on this task. The code for generating the corpus and reproducing the baselines is freely available at https://codeberg.org/haining/cctaa. Keywords: authorship identification, authorship attribution, stylometry, reproducibility, Chinese 1. Introduction Authorship attribution attempts to infer the authorship of an unsigned, single-author document by analyzing candidate authors’ writing style. It has wide applica- tions in fields such as literary history, intellectual his- tory, and online forensics. In this study, we propose a standard authorship attribution testbed for contem- porary Chinese prose. The Chinese Cross-Topic Au- thorship Attribution (CCTAA) corpus features banning content shortcuts and supporting reproducibility. 1.1. Backgrounds Problematic content shortcuts Recent work in au- thorship attribution has witnessed the introduction of a variety of new models, including models featuring deep neural networks. Although these models report impres- sive results, most of these models are evaluated using questionable test corpora. Often authors in test corpora tend to write about a limited set of topics. For exam- ple, a user, who we will refer to as “John”, in the IMDb corpus may only comment on action movies. A model trained on both content and function words is virtually certain to learn an association between John writing a document and a document featuring action-movie- specific words. This association may be sufficiently strong—if, say, no other user writes about the partic- ular kind of action movie John prefers—that a model may “learn” nothing about John’s writing style that will facilitate authorship attribution of other unsigned doc- uments by John. The model may work well on John’s writings in the IMDb corpus but may break down when presented with a blog post by John reflecting on cur- rent events. Ideally, test corpora should be designed to prevent models from leveraging topical information. Particularly inappropriate are corpora where topic or role-specific words are reliably correlated with partic- ular writers (e.g., the Enron email and IMDb corpora). Findings from recent papers illustrate the mentioned concern. Zhu and Jurgens (2021) found good perfor- mance when using only content words on ad hoc Ama- zon and Reddit corpora with a model based on pre- …

Content truncated to 500 words