An Empirical Comparison of Question Classification Methods for Question Answering Systems

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 5408–5416 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 5408 An Empirical Comparison of Question Classification Methods for Question Answering Systems Eduardo Gabriel Cortes1, Vinicius Woloszyn2, Arne Binder3, Tilo Himmelsbach2, Dante Barone1, and Sebastian M¨oller2, 3 1Federal University of Rio Grande do Sul, Porto Alegre, Brazil 2Technische Universitat Berlin, Belin, Germany 3German Research Center for Artificial Intelligence (DFKI), Berlin, Germany egcortes@inf.ufrgs.br, woloszyn@tu-berlin.de, arne.binder@dfki.de, tilo.himmelsbach@tu-berlin.de, barone@inf.ufrgs.br, sebastian.moeller@tu-berlin.de Abstract Question classification is an important component of Question Answering Systems responsible for identifying the type of an answer a particular question requires. For instance, “Who is the prime minister of the United Kingdom?” demands a name of a PERSON, while “When was the queen of the United Kingdom born?” entails a DATE. This work makes an extensible review of the most recent methods for Question Classification, taking into consideration their applicability in low-resourced languages. First, we propose a manual classification of the current state-of-the-art methods in four distinct categories: low, medium, high, and very high level of dependency on external resources. Second, we applied this categorization in an empirical comparison in terms of the amount of data necessary for training and performance in different languages. In addition to complementing earlier works in this field, our study shows a boost on methods relying on recent language models, overcoming methods not suitable for low-resourced languages. Keywords: Question Answering, Question Classification, Linguistic Resource 1. Introduction Question Answering (QA) is a field of Computer Science that aims autonomously to answer in a precise way a ques- tion posed by a user in natural language. A typical architec- ture of a QA system is composed by three fundamental el- ements (Jurafsky and Martin, 2014; Dulceanu et al., 2018; Kalouli et al., 2018): I) Question Processing: which in- tents to extract the meaning of the question; II) Information Retrieval: that retrieves relevant documents and sentences from a knowledge base; and finally III) Answer processing: that formulates the final answer. One important component in Question Processing is Ques- tion Classification, which determines the type of answer a question needs (Cortes et al., 2018). For example, the ques- tion “When will be the Easter holiday?” requires a date, while “How to make a carrot cake?” expects a recipe. The class of answer plays an important role in Questions An- swering Systems since it defines what type of information must be recovered from a knowledge base. The linguistics resources applied to Question Classifica- tion, as well as in other Machine Learning tasks are typ- ically monolingual and commonly available for a handful different languages. However, according to Ethnologue 1, nowadays there are more than 7000 languages in the world, of which 80% have fewer than a million speakers, which means that approximately six in ten people on Earth use low-resource languages. In this paper, we present an up- dated review of the state-of-the-art methods addressed to Question Classification, taking into consideration the ap- plicability …

Content truncated to 500 words