Transfer Learning from Transformers to Fake News Challenge Stance Detection (FNC-1) Task

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 1211–1218 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 1211 Transfer Learning from Transformers to Fake News Challenge Stance Detection (FNC-1) Task Valeriya Slovikovskaya, Giuseppe Attardi University of Pisa https://www.unipi.it v.slovikovskaya@studenti.unipi.it, attardi@di.unipi.it Abstract Transformer models, trained and publicly released over the last couple of years, have proved effective in many NLP tasks. We wished to test their usefulness in particular on the stance detection task. We performed experiments on the data from the Fake News Challenge Stage 1 (FNC-1). We were indeed able to improve the reported SotA on the challenge, by exploiting the generalization power of large language models based on Transformer architecture. Specifically (1) we improved the FNC-1 best performing model adding BERT sentence embedding of input sequences as a model feature, (2) we fine-tuned BERT, XLNet, and RoBERTa transformers on FNC-1 extended dataset and obtained state-of-the-art results on FNC-1 task. Keywords: fake news detection, transfer learning, fine-tuning 1. Introduction Two years ago, the Fake News Challenge Stage 1 (FNC-1) attracted the attention of several members of the linguistics research community: fifty teams participated and submit- ted their results and some of them like (Riedel et al., 2017) released their code and published research papers. Later, (Hanselowski et al., 2018) retrospectively examined the top submissions1, challenge task formulation, dataset charac- teristics as well as baseline model and evaluation metric proposed by organizers. Their contribution included re- vised FNC score, extended dataset and novel classification model that outperforms the winner’s system on class-wise F1 scores. Besides, they also determined upper bound for FNC-1 data classification conducting an annotation exper- iment that involved five human raters, who manually la- belled 200 data instances. This detailed analysis of the task, together with the code released by the authors, are highly valuable as a baseline on which we can rely in order to ob- tain further improvements. In the last two years since Fake News Challenge Cup, sig- nificant improvements have occurred in NLP technology, in particular with the development of large language mod- els using contextualized word embeddings based on the Google Transformer architecture (Vaswani et al., 2017). Applying transfer learning techniques to these general pur- pose pre-trained models helped to achieve improvements in a wide range of downstream tasks. Using these models as pre-training for downstream tasks allows lowering the amount of annotated data required for such tasks. BERT (Devlin et al., 2018), GPT (Radford et al., 2019), XLNet (Yang et al., 2019), trained on the huge unla- beled datasets on GPUs and Cloud TPUs, attained signif- icant accuracy increase on GLUE (Wang et al., 2018) and RACE (Lai et al., 2017) benchmarks2. The public releases of pre-trained models led to intensi- ond1including that of Hanselowski team Athene which ranked sec- 2GLUE leaderboard: https://gluebenchmark.com/leaderboard/, RACE leaderboard: http://www.qizhexie.com/data/ RACE_leaderboard.html fied experimenting with transfer learning / fine-tuning for a broader range of linguistic tasks; this stimulated the de- velopment of APIs that unify and …

Content truncated to 500 words