Knowledge Enhanced Masked Language Model for Stance Detection

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4725–4735 June 6–11, 2021. ©2021 Association for Computational Linguistics 4725 Knowledge Enhanced Masked Language Model for Stance Detection Kornraphop Kawintiranon and Lisa Singh Department of Computer Science Georgetown University Washington, DC, USA {kk1155,lisa.singh}@georgetown.edu Abstract Detecting stance on Twitter is especially chal- lenging because of the short length of each tweet, the continuous coinage of new termi- nology and hashtags, and the deviation of sen- tence structure from standard prose. Fine- tuned language models using large-scale in- domain data have been shown to be the new state-of-the-art for many NLP tasks, including stance detection. In this paper, we propose a novel BERT-based fine-tuning method that en- hances the masked language model for stance detection. Instead of random token masking, we propose using a weighted log-odds-ratio to identify words with high stance distinguisha- bility and then model an attention mechanism that focuses on these words. We show that our proposed approach outperforms the state of the art for stance detection on Twitter data about the 2020 US Presidential election. 1 Introduction Stance detection refers to the task of classifying a piece of text as either being in support, oppo- sition, or neutral towards a given target. While this type of labeling is useful for a wide range of opinion research, it is particularly important for un- derstanding the public’s perception of given targets, for example, candidates during an election. For this reason, our focus in this paper is on detecting stance towards political entities, namely Joe Biden and Donald Trump during the 2020 US Presidential election. Stance detection is related to, but distinct from the task of sentiment analysis, which aims to ex- tract whether the general tone of a piece of text is positive, negative, or neutral. Sobhani and col- leagues (Sobhani et al., 2016) show that measures of stance and sentiment are only 60% correlated. For example, the following sample tweet1 has an 1All of the sample tweets in this paper are invented by the authors. They are representative of real data, but do not obvious positive sentiment, but an opposing stance towards Donald Trump. I’m so happy Biden beat Trump in the debate. Stance detection is an especially difficult prob- lem on Twitter. A large part of this difficulty comes from the fact that Twitter content is short, highly dynamic, continually generating new hash- tags and abbreviations, and deviates from standard prose sentence structure. Recently, learning mod- els using pre-training (Peters et al., 2018; Radford et al., 2018; Devlin et al., 2019; Yang et al., 2019) have shown a strong ability to learn semantic rep- resentation and outperform many state-of-the-art approaches across different natural language pro- cessing (NLP) tasks. This is also true for stance detection. The strongest models for stance detec- tion on Twitter use pre-trained BERT (Ghosh et al., 2019; Sen et al., 2018). A recent study that proposed models for senti- ment analysis (Tian et al., …

Content truncated to 500 words