Question Similarity in Community Question Answering: A Systematic Exploration of Preprocessing Methods and Models

Proceedings of Recent Advances in Natural Language Processing, pages 593–601, Varna, Bulgaria, Sep 2–4, 2019. https://doi.org/10.26615/978-954-452-056-4_070 593 Question Similarity in Community Question Answering: A Systematic Exploration of Preprocessing Methods and Models Florian KunnemanA, Thiago Castro FerreiraB, Emiel KrahmerB, and Antal van den BoschA,C ACentre for Language Studies, Radboud University, The Netherlands BTilburg center for Cognition and Communication, Tilburg University, The Netherlands CKNAW Meertens Institute, Amsterdam, The Netherlands Abstract Community Question Answering forums are popular among Internet users, and a basic problem they encounter is trying to find out if their question has already been posed before. To address this issue, NLP researchers have developed methods to automatically detect question-similarity, which was one of the shared tasks in Se- mEval. The best performing systems for this task made use of Syntactic Tree Ker- nels or the SoftCosine metric. However, it remains unclear why these methods seem to work, whether their performance can be improved by better preprocessing methods and what kinds of errors they (and other methods) make. In this paper, we there- fore systematically combine and compare these two approaches with the more tradi- tional BM25 and translation-based mod- els. Moreover, we analyze the impact of preprocessing steps (lowercasing, sup- pression of punctuation and stop words re- moval) and word meaning similarity based on different distributions (word transla- tion probability, Word2Vec, fastText and ELMo) on the performance of the task. We conduct an error analysis to gain in- sight into the differences in performance between the system set-ups. The imple- mentation is made publicly available.1 1 Introduction Community Question Answering (CQA) forums, such as Quora2 and Yahoo Answers3, are popu- lar outlets to ask questions and receive answers, as 1https://github.com/fkunneman/ DiscoSumo/tree/master/ranlp 2https://www.quora.com/ 3https://answers.yahoo.com/ well as to browse through questions and answers. Given the large amount of material on these plat- forms, a basic problem users encounter is trying to find out if (a variant of) their question has already been posed (and possibly answered) before. Given a target question provided by a user, the automatic task of querying and ranking semantically simi- lar, relevant alternative questions in CQA forums is called question similarity. For efficiency reasons, the question similarity task (also known as question relevance) normally works in two ranking steps. Given a target ques- tion, the first step consists of retrieving relevant questions using a general information retrieval technique, such as BM25 (Robertson et al., 2009), or a search engine such as Google (Page et al., 1999). The second step, the focus of this work and of most other studies in this field, consists of re- ranking the most likely candidate questions with a more fine-grained, domain-specific approach. Op- tionally, the system could also return whether a candidate question is a duplicate of the query. The reranking task has been included as a benchmark task (Task 3 - Subtask B) in SemEval- 2016/2017 (Nakov et al., 2016, 2017). Using the domain of Qatar Living4, it consisted of re-ranking ten candidate questions retrieved by Google for a target question. Several …

Content truncated to 500 words