A Benchmark Corpus for the Detection of Automatically Generated Text in Academic Publications

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 4692–4700 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 4692 A Benchmark Corpus for the Detection of Automatically Generated Text in Academic Publications Vijini Liyanage, Davide Buscaldi, Adeline Nazarenko Universit´e Sorbonne Paris Nord, LIPN, CNRS, UMR 7030 F-93430, Villetaneuse, France {liyanage, davide.buscaldi, adeline.nazarenko}@lipn.univ-paris13.fr Abstract Automatic text generation based on neural language models has achieved performance levels that make the generated text almost indistinguishable from those written by humans. Despite the value that text generation can have in various applications, it can also be employed for malicious tasks. The diffusion of such practices represent a threat to the quality of academic publishing. To address these problems, we propose in this paper two datasets comprised of artificially generated research content: a completely synthetic dataset and a partial text substitution dataset. In the first case, the content is completely generated by the GPT-2 model after a short prompt extracted from original papers. The partial or hybrid dataset is created by replacing several sentences of abstracts with sentences that are generated by the Arxiv-NLP model. We evaluate the quality of the datasets comparing the generated texts to aligned original texts using fluency metrics such as BLEU and ROUGE. The more natural the artificial texts seem, the more difficult they are to detect and the better is the benchmark. We also evaluate the difficulty of the task of distinguishing original from generated text by using state-of-the-art classification models. Keywords: Automatic Text Generation, GPT-2, Arxiv-NLP, Detection, Dataset, Classification 1. Introduction The Transformer model (Vaswani et al., 2017) can be considered as a major milestone in the domain of deep learning and Natural Language Processing (NLP). Since then, various forms of state-of-the-art transformer models such as the Generative Pre-training model (GPT) (Radford et al., 2018), BERT (Devlin et al., 2018) and Transformer-XL (Dai et al., 2019) have been introduced and utilized for a diverse amount of NLP tasks. To cite some, Natural Language Gen- eration (NLG) (Radford et al., 2019), text classifica- tion (Yang et al., 2019), machine translation (Lample and Conneau, 2019) and text summarization (Lewis et al., 2019). The aforementioned research show that the transformer models are capable of producing outstand- ing results. The GPT model has been recently updated to signifi- cantly improve the quality of the generated text. For example, the GPT-2 model (Radford et al., 2019) is competent enough to produce natural text that looks like as if written by a human (Solaiman et al., 2019). Regardless of the valuable contribution provided by these models for the betterment of NLP in general, some concerns have been raised about the a potential risks associated to such models. These models can be misused for malicious tasks such as fake news gen- eration (Zellers et al., 2019), (Vosoughi et al., 2018), fake review generation (Adelani et al., 2020) and viral story generation (Faris et al., 2017), (Wardle and Der- akhshan, 2017). Recently, some of these models have …

Content truncated to 500 words