Mintaka: A Complex, Natural, and Multilingual Dataset for End-to-End Question Answering

Proceedings of the 29th International Conference on Computational Linguistics, pages 1604–1619 October 12–17, 2022. 1604 Mintaka: A Complex, Natural, and Multilingual Dataset for End-to-End Question Answering Priyanka Sen Amazon Alexa AI Cambridge, UK sepriyan@amazon.com Alham Fikri Aji Amazon Alexa AI Cambridge, UK afaji@amazon.com Amir Saffari Amazon Alexa AI Cambridge, UK amsafari@amazon.com Abstract We introduce MINTAKA, a complex, natural, and multilingual dataset designed for experi- menting with end-to-end question-answering models. Mintaka is composed of 20,000 question-answer pairs collected in English, an- notated with Wikidata entities, and translated into Arabic, French, German, Hindi, Italian, Japanese, Portuguese, and Spanish for a total of 180,000 samples. Mintaka includes 8 types of complex questions, including superlative, intersection, and multi-hop questions, which were naturally elicited from crowd workers. We run baselines over Mintaka, the best of which achieves 38% hits@1 in English and 31% hits@1 multilingually, showing that ex- isting models have room for improvement. We release Mintaka at https://github. com/amazon-research/mintaka. 1 Introduction Question answering (QA) is the task of learning to predict answers to questions. Approaches to question answering include knowledge graph (KG) based methods, which use structured data to find the correct answer (Miller et al., 2016; Saxena et al., 2020); machine reading comprehension methods, which extract answers from input documents (Ra- jpurkar et al., 2016; Kwiatkowski et al., 2019); open domain methods, which learn to retrieve rel- evant documents and extract or generate answers (Zhu et al., 2021), and closed-book methods, which use knowledge implicitly stored in model parame- ters to answer questions (Roberts et al., 2020). With state-of-the-art techniques, QA models can achieve high performance on simple questions (Shi et al., 2020, 2021) that require a single fact lookup in either a knowledge graph or a text document (e.g., "Where was Natalie Portman born?"). How- ever not all questions are simple in real-world ap- plications. We define complex questions (Lan et al., 2021) as questions that require an operation beyond a single fact lookup, such as multi-hop, compar- ative, or set intersection questions. For example, "What movie had a higher budget, Titanic or Men in Black?" requires looking up the budget of two movies, comparing the values, and selecting the movie with the higher budget. Handling more com- plex questions remains an open problem. One of the challenges in measuring and improv- ing QA performance on complex questions is a lack of datasets. While several QA datasets exist, they have shortcomings of being either large but sim- ple, such as SimpleQuestions (Bordes et al., 2015), or complex but small, such as ComplexQuestions (Bao et al., 2016) or QALD (Usbeck et al., 2018). Recently, several large and complex datasets have been released, including KQA Pro (Shi et al., 2020) and GrailQA (Gu et al., 2021). These datasets use automatically generated questions followed by hu- man paraphrasing, which can result in less natural questions, such as "Is the WOEID of Tuscaloosa 14605?" (KQA Pro) or "1520.0 is the minimum width for which size rail gauge?" (GrailQA). This can lead to a mismatch between training …

Content truncated to 500 words