QSTS: A Question-Sensitive Text Similarity Measure for Question Generation

Proceedings of the 29th International Conference on Computational Linguistics, pages 3835–3846 October 12–17, 2022. 3835 QSTS: A Question-Sensitive Text Similarity Measure for Question Generation Sujatha Das Gollapalli and See-Kiong Ng Institute of Data Science National University of Singapore {idssdg,seekiong}@nus.edu.sg Abstract While question generation (QG) has received significant focus in conversation modeling and text generation research, the problems of com- paring questions and evaluation of QG models have remained inadequately addressed. Indeed, QG models continue to be evaluated using tradi- tional measures such as BLEU, METEOR, and ROUGE scores which were designed for other text generation problems. We propose QSTS, a novel Question-Sensitive Text Similarity mea- sure for questions that characterizes their target intent based on question class, named-entity, and semantic similarity information. We show that QSTS addresses several short- comings of existing measures that depend on n-gram overlap scores and obtains superior results compared to traditional measures on publicly-available QG datasets. We also collect a novel dataset SimQG for enabling question similarity research in QG contexts. SimQG contains questions generated by state-of-the-art QG models along with human judgements on their relevance with respect to passage contexts as well as the given reference questions. Using SimQG, we showcase the key aspect of QSTS that differentiates it from all existing measures. QSTS is not only able to characterize similar- ity between two questions, but is also able to score questions with respect to passage con- texts. Thus QSTS is, to our knowledge, the first metric that enables the measurement of QG performance in a reference-free manner. 1 Introduction Automatic Question Generation (QG), the task of generating natural language questions for a given input text passage continues to garner significant research focus in the NLP community (Wang et al., 2020b; Huang et al., 2021) due its potential applica- tion in education (Srivastava and Goodman, 2021), tutoring (Lindberg et al., 2013) and interactive dia- log systems (Wang et al., 2020a). In current research, in lieu of human evalua- tion, the standard practice for evaluating the perfor- mance of QG models involves the use of Question Answering (QA) datasets containing pairs of (ref- erence question, passage context) elements. For evaluating QG, the machine-generated question for a given passage context is compared with the given reference question by applying metrics such as BLEU (Papineni et al., 2002), METEOR Lavie and Agarwal (2007), and ROUGE (Lin, 2004). The above widely-used measures were originally developed for evaluating tasks such as summariza- tion and translation and are based on overlap of n-grams between a given reference text and the model-generated text. Though studies have indi- cated that these measures do not correlate well with human judgements of fluency, relevance, and co- herence (Callison-Burch et al., 2006; Liu et al., 2016; Nema and Khapra, 2018) these measures are easy to compute and continue to be used for various natural language generation (NLG) tasks. Recently though, research studies are addressing metrics learning for NLG using transformers and these learnt metrics were shown to obtain state- of-the-art performance in evaluation (Zhang et …

Content truncated to 500 words