Elvis vs. M. Jackson: Who has More Albums? Classification and Identification of Elements in Comparative Questions

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 3771–3779 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 3771 Elvis vs. M. Jackson: Who has More Albums? Classification and Identification of Elements in Comparative Questions Meriem Beloucif1, Seid Muhie Yimam2, Steffen Stahlhacke2, Chris Biemann2 1 Department of Linguistics and Philology, Uppsala University, Sweden 2Universit¨at Hamburg, Department of Informatics, Language Technology group, Germany 1firstname.lastname@lingfil.uu.se, 2firstname.lastname@uni-hamburg.de Abstract Comparative Question Answering (cQA) is the task of providing concrete and accurate responses to queries such as: “Is Lyft cheaper than a regular taxi?” or “What makes a mortgage different from a regular loan?”. In this paper, we propose two new open-domain real-world datasets for identifying and labeling comparative questions. While the first dataset contains instances of English questions labeled as comparative vs. non-comparative, the second dataset provides additional labels including the objects and the aspects of comparison. We conduct several experiments that evaluate the soundness of our datasets. The evaluation of our datasets using various classifiers show promising results that reach close-to-human results on a binary classification task with a neural model using ALBERT embeddings. When approaching the unsupervised sequence labeling task, some headroom remains. Keywords: Comparative Question Answering, Labeling, Classification 1. Introduction Most Question Answering Systems (QAS) research fo- cuses on answering factoid questions, but fails at an- swering comparative questions in an efficient argumen- tative manner. Comparative questions are generally treated the same as any open domain question, although they have a different taxonomy. Therefore, we start in this paper by conducting a linguistic study about the taxonomy of comparative sentences and questions. Next, our goal is to analyze and identify a set of real- world questions submitted with comparison intent. In websites like Yahoo! Answers1, Quora2 or Reddit3, there is a whole range of comparative questions in terms of topics, language difficulty, and format. For instance, What is the difference between a cappuccino and a latte? is a simple comparative question, whereas Why is it that hot water cleans better than cold wa- ter, when washing a jeans? is much more complex. In this paper, we propose a thorough study on compar- ative questions, and propose two manually annotated datasets for comparative Question Answering (cQA). More specifically, we tackle two tasks: 1) a binary clas- sification task for assessing if a question is comparative or not; and 2) a sequence labeling task, where we iden- tify the elements of comparison in a supervised and un- supervised manners. a. Is Athens small compared to Rome b. Is [ Object 1 Athens] [ Aspect small] com- pared to [ Object 2 Rome] The identification of elements consists of labeling the 1Yahoo! Answers has been dissolved in May 2021: https://answers.yahoo.com/ 2https://www.quora.com/ 3https://www.reddit.com/ objects of comparison and the comparative aspect; for instance, in the example above, Athens is Object 1 and small is the Aspect. Comparative elements identifica- tion is the first step towards cQA understanding and an- swering. In this work, we purposefully focus our …

Content truncated to 500 words