ComQA: A Community-sourced Dataset for Complex Factoid Question Answering with Paraphrase Clusters

Proceedings of NAACL-HLT 2019, pages 307–317 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 307 ComQA: A Community-sourced Dataset for Complex Factoid Question Answering with Paraphrase Clusters Abdalghani Abujabal1, Rishiraj Saha Roy2, Mohamed Yahya3 and Gerhard Weikum2 1Amazon Alexa, Aachen, Germany abujabaa@amazon.de 2Max Planck Institute for Informatics, Saarland Informatics Campus, Germany {rishiraj, weikum}@mpi-inf.mpg.de 3Bloomberg L.P., London, United Kingdom yahya.mohamed@gmail.com Abstract To bridge the gap between the capabilities of the state-of-the-art in factoid question an- swering (QA) and what users ask, we need large datasets of real user questions that cap- ture the various question phenomena users are interested in, and the diverse ways in which these questions are formulated. We introduce ComQA, a large dataset of real user questions that exhibit different challenging aspects such as compositionality, temporal reasoning, and comparisons. ComQA questions come from the WikiAnswers community QA platform, which typically contains questions that are not satisfactorily answerable by existing search engine technology. Through a large crowd- sourcing effort, we clean the question dataset, group questions into paraphrase clusters, and annotate clusters with their answers. ComQA contains 11, 214 questions grouped into 4,834 paraphrase clusters. We detail the process of constructing ComQA, including the measures taken to ensure its high quality while mak- ing effective use of crowdsourcing. We also present an extensive analysis of the dataset and the results achieved by state-of-the-art systems on ComQA, demonstrating that our dataset can be a driver of future research on QA. 1 Introduction Factoid QA is the task of answering natural lan- guage questions whose answer is one or a small number of entities (Voorhees and Tice, 2000). To advance research in QA in a manner consistent with the needs of end users, it is important to have access to datasets that reflect real user infor- mation needs by covering various question phe- nomena and the wide lexical and syntactic vari- ety in expressing these information needs. The 1The main part of this work was carried out when the au- thor was at the Max Planck Institute for Informatics. A: [https://en.wikipedia.org/wiki/cairo] Q: “largest city located along the Nile river?” Q: “largest city by the Nile river?” Q: “What is the largest city in Africa that is on the banks of the Nile river?” Cluster 2 comparison Q: “Who was the Britain’s leader during WW1?” Q: “Who ran Britain during WW1?” Q: “Who was the leader of Britain during World War One?” Cluster 1 A: [https://en.wikipedia.org/wiki/h._h._asquith, https://en.wikipedia.org/wiki/david_lloyd_george] temporal Q: “John Travolta and Jamie Lee Curtis acted in this film?” Q: “Jamie Lee Curtis and John Travolta played together in this movie?” Q: “John Travolta and Jamie Lee Curtis were actors in this film?” Cluster 3 A: [https://en.wikipedia.org/wiki/perfect_(film) compositional Q: “Who is the first human landed in Mars?” Q: “Who was the first human being on Mars?” Q: “first human in Mars?” Cluster 4 A: [] empty answer Figure 1: ComQA paraphrase clusters covering a range of question aspects e.g., temporal and compositional questions, with lexical and syntactic …

Content truncated to 500 words