Detecting Semantically Equivalent Questions in Online User Forums

Proceedings of the 19th Conference on Computational Language Learning, pages 123–131, Beijing, China, July 30-31, 2015. c⃝2015 Association for Computational Linguistics Detecting Semantically Equivalent Questions in Online User Forums Dasha Bogdanova∗, C´ıcero dos Santos†, Luciano Barbosa† and Bianca Zadrozny† ∗ADAPT centre, School of Computing, Dublin City University, Dublin, Ireland dbogdanova@computing.dcu.ie †IBM Research, 138/146 Av. Pasteur, Rio de Janeiro, Brazil {cicerons,lucianoa,biancaz}@br.ibm.com Abstract Two questions asking the same thing could be too different in terms of vocabulary and syntactic structure, which makes identify- ing their semantic equivalence challeng- ing. This study aims to detect semanti- cally equivalent questions in online user forums. We perform an extensive number of experiments using data from two differ- ent Stack Exchange forums. We compare standard machine learning methods such as Support Vector Machines (SVM) with a convolutional neural network (CNN). The proposed CNN generates distributed vec- tor representations for pairs of questions and scores them using a similarity metric. We evaluate in-domain word embeddings versus the ones trained with Wikipedia, estimate the impact of the training set size, and evaluate some aspects of do- main adaptation. Our experimental re- sults show that the convolutional neural network with in-domain word embeddings achieves high performance even with lim- ited training data. 1 Introduction Question-answering (Q&A) community sites, such as Yahoo! Answers,1 Quora2 and Stack Ex- change,3 have gained a lot of attention in the recent years. Most Q&A community sites advise users to search the forum for an answer before posting a new question. However, this is not always an easy task because different users could formulate the same question in completely different ways. Some user forums, such as those of the Stack Exchange online community, have a duplication policy. Ex- act duplicates, such as copy-and-paste questions, 1 https://answers.yahoo.com/ 2 http://www.quora.com 3 http://stackexchange.com/ and nearly exact duplicates are usually quickly de- tected, closed and removed from the forum. Nev- ertheless, some duplicate questions are kept. The main reason for that is that there are many ways to ask the same question, and a user might not be able to find the answer if they are asking it a dif- ferent way.4 In this study we define two questions as seman- tically equivalent if they can be adequately an- swered by the exact same answer. Table 1 presents an example of a pair of such questions from Ask Ubuntu forum. Detecting semantically equivalent questions is a very difficult task due to two main factors: (1) the same question can be rephrased in many different ways; and (2) two questions could be asking different things but look for the same solution. Therefore, traditional similarity mea- sures based on word overlap such as shingling and Jaccard coefficient (Broder, 1997) and its varia- tions (Wu et al., 2011) are not able to capture many cases of semantic equivalence. In this paper, we propose a convolutional neural network architecture to detect semantically equiv- alent questions. The proposed CNN first trans- forms words into word embeddings (Mikolov et al., 2013), using a large collection …

Content truncated to 500 words