The Dark Side of the Language: Pre-trained Transformers in the DarkNet

Proceedings of Recent Advances in Natural Language Processing, pages 949–960 Varna, Sep 4–6, 2023 https://doi.org/10.26615/978-954-452-092-2_102 949 The Dark Side of the Language: Pre-trained Transformers in the DarkNet Leonardo Ranaldi (∗), Aria Nourbakhsh(∗), Arianna Patrizi(∗), Elena Sofia Ruzzetti(∗), Dario Onorati(•), Michele Mastromattei(∗), Francesca Fallucchi(∗), Fabio Massimo Zanzotto(∗) (∗) Human-Centric ART Group, Department of Enterprise Engineering, University of Rome Tor Vergata. (•) Department of Computer, Control and Management Engineering, University of Rome "Sapienza". [first name].[last name]@uniroma2.it Abstract Pre-trained Transformers are challenging hu- man performances in many NLP tasks. The massive datasets used for pre-training seem to be the key to their success on existing tasks. In this paper, we explore how a range of pre- trained Natural Language Understanding mod- els perform on definitely unseen sentences pro- vided by classification tasks over a DarkNet corpus. Surprisingly, results show that syn- tactic and lexical neural networks perform on par with pre-trained Transformers even after fine-tuning. Only after what we call extreme domain adaptation, that is, retraining with the masked language model task on all the novel corpus, pre-trained Transformers reach their standard high results. This suggests that huge pre-training corpora may give Transformers un- expected help since they are exposed to many of the possible sentences. 1 Introduction Transformers (Zhang et al., 2019; Radford and Narasimhan, 2018) have been rocking the field of NLP. These Transformers are outperforming all previous methods and, sometimes, even humans in many NLP tasks (Wang et al., 2018, 2020; Kalyan et al., 2021; Guo et al., 2022). Pre-training on large corpora seems to be the key that boosts performances, as these Transformers may induce clear models of target languages. In- deed, BERT is pre-trained on an English corpus of 3,300M words consisting of books (Zhu et al., 2015a) and Wikipedia. The English version of the last ERNIE (Sun et al., 2021a) is trained on an even more extensive corpus. MEGATRON-LM (Shoeybi et al., 2019) utilizes an incredible cor- pus of 174 GB, and the Chinese version of ERNIE breaks the records by exploiting a 4TB corpus (Sun et al., 2021b). Therefore, the challenge is training over always more massive corpora. Huge pre-training corpora may give unexpected help to Transformers: their successes in down- stream tasks can be because Transformers have seen large parts of possible sentences. This could be a sort of overfitting. This possible shortcoming is sometimes considered when novel Transformers are introduced (Radford et al., 2019; Shoeybi et al., 2019). For this reason, Radford et al. (2019) have excluded Wikipedia pages for pre-training as it is a common data source for other downstream datasets. Yet, when using off-the-shelf pre-trained models, this caution is generally disregarded. For exam- ple, the discovering ongoing conversation (DOC) task, introduced by Zanzotto and Ferrone (2017) was found challenging for humans, but the BERT baseline model achieved the astonishing 88.4 F1 score (Wang et al., 2020). DOC consists of deter- mining if two utterances are contiguous in classical theatrical plays. These plays may be included in the book dataset (Zhu et al., …

Content truncated to 500 words