Contextualized Embeddings based Transformer Encoder for Sentence Similarity Modeling in Answer Selection Task

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 5505–5514 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 5505 Contextualized Embeddings based Transformer Encoder for Sentence Similarity Modeling in Answer Selection Task Md Tahmid Rahman Laskar1, 3, Jimmy Huang2, 3, Enamul Hoque2 1 Department of Electrical Engineering and Computer Science, York University 2 School of Information Technology, York University 3 Information Retrieval & Knowledge Management Research Lab, York University Toronto, Ontario, Canada tahmedge@cse.yorku.ca, jhuang@yorku.ca, enamulh@yorku.ca Abstract Word embeddings that consider context have attracted great attention for various natural language processing tasks in recent years. In this paper, we utilize contextualized word embeddings with the transformer encoder for sentence similarity modeling in the answer selection task. We present two different approaches (feature-based and fine-tuning-based) for answer selection. In the feature-based approach, we utilize two types of contextualized embeddings, namely the Embeddings from Language Models (ELMo) and the Bidirectional Encoder Representations from Transformers (BERT) and integrate each of them with the transformer encoder. We find that integrating these contextual embeddings with the transformer encoder is effective to improve the performance of sentence similarity modeling. In the second approach, we fine-tune two pre-trained transformer encoder models for the answer selection task. Based on our experiments on six datasets, we find that the fine-tuning approach outperforms the feature-based approach on all of them. Among our fine-tuning-based models, the Robustly Optimized BERT Pretraining Approach (RoBERTa) model results in new state-of-the-art performance across five datasets. Keywords: Answer Selection, Transformer Encoder, Contextualized Embeddings, ELMo, BERT, RoBERTa, Deep Learning 1. Introduction Measuring the similarity between question answering pairs (Yih et al., 2013) is a fundamental problem in the areas of Information Retrieval and Natural Language Pro- cessing (NLP). For example, in the answer selection task, a question along with list of candidate answers are given and then the answers are ranked based on their relevance with the question (see Table 1). Several neural models for the an- swer selection task use word embeddings like GloVe (Pen- nington et al., 2014) or Word2Vec (Mikolov et al., 2013) as input and measure the similarity between the vector repre- sentation of two sentences produced by the models (Chen et al., 2018a; Chen et al., 2018b). However, such embed- dings provide fixed representation for each word and do not capture its context in different sentences. Recently, contextualized word representation methods such as ELMo and BERT received a lot of attention. The ELMo model (Peters et al., 2018) learns the contextualized word representations from a deep bidirectional language model pre-trained on large text corpora. The BERT model (Devlin et al., 2019) can also generate contextual embeddings like ELMo via utilizing the encoder of transformer (Vaswani et al., 2017) and yields very good results on tasks such as named-entity recognition. Since these contextual em- beddings can capture better representation of a sentence by generating embedding of each word based on its surround- ing context, we are motivated to use them for sentence sim- ilarity modeling …

Content truncated to 500 words