Ways of Asking and Replying in Duplicate Question Detection

Proceedings of the 6th Joint Conference on Lexical and Computational Semantics (*SEM 2017), pages 262–270, Vancouver, Canada, August 3-4, 2017. c⃝2017 Association for Computational Linguistics Ways of Asking and Replying in Duplicate Question Detection Jo˜ao Rodrigues, Chakaveh Saedi, Vladislav Maraev, Jo˜ao Silva, Ant´onio Branco University of Lisbon {joao.rodrigues,chakaveh.saedi,vlad.maraev, jsilva,antonio.branco}@di.fc.ul.pt Abstract This paper presents the results of sys- tematic experimentation on the impact in duplicate question detection of different types of questions across both a number of established approaches and a novel, supe- rior one used to address this language pro- cessing task. This study permits to gain a novel insight on the different levels of robustness of the diverse detection meth- ods with respect to different conditions of their application, including the ones that approximate real usage scenarios. 1 Introduction Automatic detection of semantically equivalent questions is a language processing task of the ut- most importance given the upsurge of interest in conversational interfaces. It is a key procedure in finding answers to questions. For instance, in a context of customer support via a chat channel, with the help of duplicate question detection, pre- vious interactions between customers and human operators can be explored to provide an increas- ingly automatic question answering service. If a new input question is equivalent to a question al- ready stored, it can be replied automatically with the answer stored with its recorded duplicate. Though it has been less researched than simi- lar tasks, duplicate question detection (DQD) is attracting an increasing interest. It can be seen as belonging to a family of semantic text sim- ilarity tasks, which have been addressed in Se- mEval challenges since 2012, and which in the last SemEval2016, for instance, included also tasks like plagiarism detection or degree of similarity between machine translation output and its post- edited version, among others. Semantic textual similarity assesses the degree to which two tex- tual segments are semantically equivalent to each other, which is typically scored on an ordinal scale ranging from semantic equivalence to complete semantic dissimilarity. Paraphrase detection can be seen as a special case of semantic textual similarity, where the scale is reduced to its two extremes and the outcome for an input pair is yes/no. DQD, in turn, could be seen as a special case of paraphrase detection that is restricted to interrogative expressions. While SemEval2016 had no task on yes/no DQD, it had a “Question-Question” graded sim- ilarity subtask of Task 1. The top performing system in this subtask (0.74 Pearson correlation) scored below the best result when all subtasks of Task 1 are considered (0.77), and also below the best scores of many of the other subtasks (e.g. 0.84 in plagiarism detection (Agirre et al., 2016)). While scores obtained for different tasks by sys- tems trained and evaluated over different datasets cannot be compared, those results nonetheless lead one to ponder whether focusing on pairs of interrogatives may be a task that is harder than paraphrase detection that focuses on pairs of non- interrogatives (e.g. plagiarism …

Content truncated to 500 words