CoNTACT: A Dutch COVID-19 Adapted BERT for Vaccine Hesitancy and Argumentation Detection

Proceedings of the 29th International Conference on Computational Linguistics, pages 6837–6845 October 12–17, 2022. 6837 CoNTACT: A Dutch COVID-19 Adapted BERT for Vaccine Hesitancy and Argumentation Detection Jens Lemmens and Jens Van Nooten and Tim Kreutz and Walter Daelemans CLiPS (University of Antwerp) Lange Winkelstraat 40, 2000 Antwerp (Belgium) Abstract We present CoNTACT1: a Dutch language model adapted to the domain of COVID-19 tweets. The model was developed by con- tinuing the pre-training phase of RobBERT (Delobelle et al., 2020) by using 2.8M Dutch COVID-19 related tweets posted in 2021. In order to test the performance of the model and compare it to RobBERT, the two models were tested on two tasks: (1) binary vaccine hesi- tancy detection and (2) detection of arguments for vaccine hesitancy. For both tasks, not only Twitter but also Facebook data was used to show cross-genre performance. In our experi- ments, CoNTACT showed statistically signif- icant gains over RobBERT in all experiments for task 1. For task 2, we observed substantial improvements in virtually all classes in all ex- periments. An error analysis indicated that the domain adaptation yielded better representa- tions of domain-specific terminology, causing CoNTACT to make more accurate classifica- tion decisions. 1 Introduction Since the development of COVID-19 vaccines, an us-against-them mentality has emerged between ad- vocates and adversaries of vaccines. Social media functions as a catalyst in this polarization, because it enables a rapid spread of unsolicited opinions, ranging from nuanced to radical, and well-reasoned to emotional. The turbulent vaccine debate that has been hosted by social media will unquestion- ably continue to influence future views on vaccina- tion, whether it regards new COVID-19 boosters or other vaccines. In order to help process the large amounts of online content related to vaccines and COVID-19 in general, we present CoNTACT (Con- textual Neural Transformer Adapted to COVID-19 Tweets). CoNTACT was developed by fine-tuning 1The model is available at https://huggingface. co/clips/contact RobBERT (Delobelle et al., 2020) (a RoBERTa- base model (Liu et al., 2019) pre-trained on Dutch data) on masked language modeling using 2.8M Dutch-language tweets related to COVID-19 that were posted in 2021. The model was evaluated on two tasks: (1) binary vaccine hesitancy classifica- tion and (2) classification of arguments for vaccine hesitancy. In order to measure the effect of the domain adaptation, the results were compared to out-of-the-box RobBERT. Moreover, the aforemen- tioned tasks were not only performed on tweets, but also on Facebook comments to show the cross- genre benefits of the domain adaptation. After- wards, a qualitative error analysis was conducted to show where CoNTACT improved compared to Rob- BERT and where it could potentially improve fur- ther. In this analysis, special attention was paid to what domain-specific language the model learned compared to out-of-the-box RobBERT. In earlier research, an English language model pre-trained on COVID-19 related tweets (COVID- Twitter-BERT) was developed (Müller et al., 2020). We apply the same methodology for the first time to Dutch and extensively test the effect on two COVID-19 related classification tasks. …

Content truncated to 500 words