ArgueBERT: How To Improve BERT Embeddings for Measuring the Similarity of Arguments

ArgueBERT: How To Improve BERT Embeddings for Measuring the Similarity of Arguments Maike Behrendt Heinrich Heine University maike.behrendt@hhu.de Stefan Harmeling Heinrich Heine University harmeling@hhu.de Abstract Argumentation is an important tool within hu- man interaction, not only in law and politics but also for discussing issues, expressing and exchanging opinions and coming to decisions in our everyday life. Applications for argu- mentation often require the measurement of the arguments’ similarity, to solve tasks like clustering, paraphrase identification or summa- rization. In our work, BERT embeddings are pre-trained on novel training objectives and af- terwards fine-tuned in a siamese architecture, similar to Reimers and Gurevych (2019b), to measure the similarity of arguments. The ex- periments conducted in our work show that a change in BERT’s pre-training process can im- prove the performance on measuring argument similarity. 1 Introduction Since today it is common to share opinions on so- cial media to discuss and argue about all kinds of topics, the interest of research in the field of artificial intelligence in argumentation is con- stantly rising. Tasks like counter-argument re- trieval (Wachsmuth et al., 2018), argument cluster- ing (Reimers et al., 2019a) and identifying the most prominent arguments in online debates (Boltuˇzi´c and ˇSnajder, 2015) have been examined and au- tomated in the past. Many of these tasks involve measuring the textual similarity of arguments. Transformer-based language models such as the bi-directional encoder representations from transformers (BERT) by Devlin et al. (2019) are widely used for different natural language process- ing (NLP) tasks. Nevertheless, for large-scale tasks like finding the most similar sentence in a collec- tion of sentences, BERT’s cross-encoding approach is disadvantageous as it creates a huge computa- tional overhead. In our work, we focus on exactly these large- scale tasks. We want to train embeddings of ar- guments in order to measure their similarity, e.g., to automatically recognize similar user entries in ongoing discussions in online argumentation sys- tems. In this way redundancy can be avoided when collecting arguments. We base our approach on Sentence-BERT (SBERT), proposed by Reimers and Gurevych (2019b), which is a bi-encoder, fine- tuning the model’s parameters to place similar sen- tences close to one another in the vector space. This approach yields good results on paraphrase identification tasks, but evaluating it on an argu- ment similarity corpus shows a noticeable drop in performance. To improve this method, we propose and evalu- ate three alternative pre-training tasks that replace the next sentence prediction (NSP) in BERT’s pre- training process to optimize SBERT for measuring the similarity of arguments. These proposed tasks are similarity prediction, argument order prediction and argument graph edge validation. Being pre- trained on these tasks and fine-tuned in a siamese SBERT architecture, we call these models argue- BERT throughout this work. To examine the models’ applicability in practice, we also propose a new evaluation task, which is called similar argument mining (SAM). Solving the task of SAM includes recognizing paraphrases (if any are present) in a large set of arguments, e.g., when a …

Content truncated to 500 words