An empirical analysis of existing systems and datasets toward general simple question answering

Proceedings of the 28th International Conference on Computational Linguistics, pages 5321–5334 Barcelona, Spain (Online), December 8-13, 2020 5321 An empirical analysis of existing systems and datasets toward general simple question answering Namgi Han♠♥♦ Goran Topi´c♠ Hiroshi Noji♠ Hiroya Takamura♠♣ Yusuke Miyao♠♭ ♠Artificial Intelligence Research Center, AIST ♥The Graduate University for Advanced Studies ♦National Institute of Informatics ♣Tokyo Institute of Technology ♭The University of Tokyo {han.namgi, goran.topic, hiroshi.noji}@aist.go.jp takamura@pi.titech.ac.jp yusuke@is.s.u-tokyo.ac.jp Abstract In this paper, we evaluate the progress of our field toward solving simple factoid questions over a knowledge base, a practically important problem in natural language interface to database. As in other natural language understanding tasks, a common practice for this task is to train and evaluate a model on a single dataset, and recent studies suggest that SimpleQuestions, the most popular and largest dataset, is nearly solved under this setting. However, this common setting does not evaluate the robustness of the systems outside of the distribution of the used training data. We rigorously evaluate such robustness of existing systems using different datasets. Our analysis, including shifting of training and test datasets and training on a union of the datasets, suggests that our progress in solving SimpleQuestions dataset does not indicate the success of more general simple question answering. We discuss a possible future direction toward this goal. 1 Introduction Simple factoid question answering over a knowledge base is an important task in natural language un- derstanding. Although it only deals with factoid questions about a single entity and a predicate, they cover much of the real user queries (Dai et al., 2016), and also, accurate mapping of these is a critical subproblem in semantic parsing-based complex query generation (Berant et al., 2013; Bao et al., 2016; Reddy et al., 2016; Trivedi et al., 2017). SimpleQuestions (Bordes et al., 2015) is the largest and most popular dataset on this task. It was recently argued that this task, given abundant training data, is nearly solved with standard techniques in machine learning (Petrochuk and Zettlemoyer, 2018; Mohammed et al., 2018). In this paper, we present a thorough empirical analysis to assess whether the success of one particular dataset indicates the success of the task itself in general. To this end, we evaluate the behaviors of four existing QA systems targeting SimpleQuestions (Mohammed et al., 2018; Yu et al., 2017; Wu et al., 2019; Huang et al., 2019), across four different datasets (Cai and Yates, 2013; Yih et al., 2016; Bordes et al., 2015; Jiang et al., 2019), under different conditions. One of our research goals is to evaluate the robustness of a model trained on a single dataset against questions that are outside of the distribution of the training data. Such robustness evaluation is recently actively studied in other language understanding tasks (Jia and Liang, 2017; Naik et al., 2018; McCoy et al., 2019; Ribeiro et al., 2020) while little effort has been made on question answering over a knowledge base, though, in practice, it would be critical because a practical system …

Content truncated to 500 words