Competence-based Question Generation

Proceedings of the 29th International Conference on Computational Linguistics, pages 1521–1533 October 12–17, 2022. 1521 Competence-based Question Generation Jingxuan Tu, Kyeongmin Rim, James Pustejovsky Department of Computer Science Brandeis University {jxtu,krim,jamesp}@brandeis.edu Abstract Models of natural language understanding of- ten rely on question answering and logical in- ference benchmark challenges to evaluate the performance of a system. While informative, such task-oriented evaluations do not assess the broader semantic abilities that humans have as part of their linguistic competence when speaking and interpreting language. We de- fine competence-based (CB) question gener- ation, and focus on queries over lexical se- mantic knowledge involving implicit argument and subevent structure of verbs. We present a method to generate such questions and a dataset of English cooking recipes we use for imple- menting the generation method. Our primary experiment shows that even large pretrained language models perform poorly on CB ques- tions until they are provided with additional contextualized semantic information. The data and the source code is available at: https: //github.com/brandeis-llc/CompQG. 1 Introduction Developing an Artificial Intelligence (AI) system with inferencing and reasoning capabilities that enables itself to communicate with intellectual hu- man users has been a holy grail of the research community. For example, there has been consid- erable effort put on large challenges in question answering formats to measure such inference and communication capabilities of neural end-to-end systems (Ribeiro et al., 2020; Prabhumoye et al., 2020; Rogers et al., 2021; Minaee et al., 2021). However, we argue that a broader effort needs to be put on measurements more focused on linguistic competencies, and not just on extractive compre- hension skills or “challenge checklisting”. Out ar- gument is in line with some moves in this direction (Johnson et al., 2017), but there is still no generally accepted distinction in current natural language pro- cessing (NLP) between challenge-based tasks and competence-based (CB) performance (Bentivogli et al., 2017). Analogous to human cognitive compe- tencies, there is both a methodological and model- ing advantage to focusing a system’s performance on competence-based learning rather than a nar- rowly defined task or challenge. The term competence-based (CB) has been ap- plied to a number of different concepts, from lin- guistics (Chomsky, 1965) to both the science of learning and educational communities (Bechtel et al., 1999; Voorhees, 2001; Chyung et al., 2006; Hsiao et al., 2020; Platanios et al., 2019). The com- mon core to both is a concept capturing a coherent set of abilities that an individual has in a specific do- main (Doignon and Falmagne, 1985; Heller et al., 2013). More recently, we proposed a new multi- modal question answering task R2VQ that focuses on querying competence-based knowledge from cooking recipe texts and videos (Tu et al., 2022). For the text part, we created a dataset with rich an- notations of hidden arguments and coreference that focuses on lexical competence as deployed in both single and multiple sentence composition (Puste- jovsky, 1995; Marconi, 1997; Geeraerts, 2009). As a natural complement and extend to the R2VQ task (Tu et …

Content truncated to 500 words