forumBERT: Topic Adaptation and Classification of Contextualized Forum Comments in German

forumBERT: Topic Adaptation and Classification of Contextualized Forum Comments in German Ayush Yadav International Institute of Information Technology Bangalore, India yadavayush.ay42@gmail.com Benjamin Milde Language Technology Group Dept. of Informatics Universit¨at Hamburg, Germany milde@informatik.uni-hamburg.de Abstract Online user comments in public forums are of- ten associated with low quality, hate speech or even excessive demands for moderation. To better exploit their constructive and deliber- ate potential, we present forumBERT. forum- BERT is built on top of the BERT architec- ture and uses a shared weight and late fusion technique to better determine the quality and relevance of a comment on a forum article. Our model integrates article context with com- ments for the online/offline comment modera- tion task. This is done using a two step proce- dure: self-supervised BERT language model fine tuning for topic adaptation followed by in- tegration into the forumBERT architecture for online/offline classification. We present evalu- ation results on various classification tasks of the public One Million Post dataset, as well as on the online/offline comment moderation task on 998,158 labelled comments from NDR.de, a popular German broadcaster’s website. fo- rumBERT significantly outperforms baseline models on the NDR dataset and also outper- forms all existing advanced baseline models on the OMP dataset. Additionally we conduct two studies on the influence of topic adaptation on the general comment moderation task. 1 Introduction Online user comments, such as those on journalis- tic content or product features are often associated with low quality, hate speech or even excessive demands for moderation. Automating this moder- ation or aspects of it can be considered to be of high practical interest. One of the key challenges of forum comment moderation is the specificity of category of classification. Forum comments have to be moderated for hate-speech, discrimination, spam among many other generally discussed clas- sification tasks. Additionally comments on forum articles must also be moderated for relevance and contribution to the discourse. Previous work by Schabus et al. (2017) and Sch- abus and Skowron (2018) introduces the idea of applied classification, wherein comments are anno- tated across multiple forum specific categories and classification models are created for each category. In this paper we focus on the more general ”com- ment moderation task” on news forum comments. In this task, comments can be classified into one of two categories, either online or offline, where an online classification represents a comment that is accepted by the forum moderators and an offline classification represents comments that have been taken down by the forum moderators. In recent years, the Natural Language Process- ing community has experienced a substantial shift towards using pre-trained models. Their usage on large corpora has proved to be beneficial in learn- ing general language representations and has shown improvement in text classification and many other NLP tasks, which has also helped avoid training large language models from scratch. However, the lack of portability of NLP models to new condi- tions is a central issue in NLP. For many target applications like comment moderation on niche …

Content truncated to 500 words