tBERT: Topic Models and BERT Joining Forces for Semantic Similarity Detection

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7047–7055 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 7047 tBERT: Topic Models and BERT Joining Forces for Semantic Similarity Detection Nicole Peinelt1,2 and Dong Nguyen1,3 and Maria Liakata1,2 1The Alan Turing Institute, London, UK 2University of Warwick, Coventry, UK 3Utrecht University, Utrecht, The Netherlands {n.peinelt, m.liakata}@warwick.ac.uk, dnguyen@turing.ac.uk Abstract Semantic similarity detection is a fundamen- tal task in natural language understanding. Adding topic information has been useful for previous feature-engineered semantic similar- ity models as well as neural models for other tasks. There is currently no standard way of combining topics with pretrained contex- tual representations such as BERT. We pro- pose a novel topic-informed BERT-based ar- chitecture for pairwise semantic similarity de- tection and show that our model improves per- formance over strong neural baselines across a variety of English language datasets. We find that the addition of topics to BERT helps par- ticularly with resolving domain-specific cases. 1 Introduction Modelling the semantic similarity between a pair of texts is a crucial NLP task with applications ranging from question answering to plagiarism de- tection. A variety of models have been proposed for this problem, including traditional feature- engineered techniques (Filice et al., 2017), hybrid approaches (Wu et al., 2017; Feng et al., 2017; Koreeda et al., 2017) and purely neural architec- tures (Wang et al., 2017; Tan et al., 2018; Deriu and Cieliebak, 2017). Recent pretrained contextu- alised representations such as ELMo (Peters et al., 2018) and BERT (Devlin et al., 2019) have led to impressive performance gains across a variety of NLP tasks, including semantic similarity detection. These models leverage large amounts of data to pretrain text encoders (in contrast to just individual word embeddings as in previous work) and have established a new pretrain-finetune paradigm. While large improvements have been achieved on paraphrase detection (Tomar et al., 2017; Gong et al., 2018), semantic similarity detection in Com- munity Question Answering (CQA) remains a chal- lenging problem. CQA leverages user-generated content from question answering websites (e.g. StackExchange) to answer complex real-world questions (Nakov et al., 2017). The task requires modelling the relatedness between question-answer pairs which can be challenging due to the highly domain-specific language of certain online forums and low levels of direct text overlap between ques- tions and answers. Topic models may provide additional signals for semantic similarity, as earlier feature-engineered models for semantic similarity detection success- fully incorporated topics (Qin et al., 2009; Tran et al., 2015; Mihaylov and Nakov, 2016; Wu et al., 2017). They could be especially useful for dealing with domain-specific language since topic models have been exploited for domain adaptation (Hu et al., 2014; Guo et al., 2009). Moreover, recent work on neural architectures has shown that the in- tegration of topics can yield improvements in other tasks such as language modelling (Ghosh et al., 2016), machine translation (Chen et al., 2016), and summarisation (Narayan et al., 2018; Wang et al., 2018). We therefore introduce a …

Content truncated to 500 words