SimQA: Detecting Simultaneous MT Errors through Word-by-Word Question Answering

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 562 - 581 December 7-11, 2022 ©2022 Association for Computational Linguistics QRelScore: Better Evaluating Generated Questions with Deeper Understanding of Context-aware Relevance Xiaoqiang Wang1∗, Bang Liu2∗†, Siliang Tang1‡ and Lingfei Wu3‡ 1Zhejiang University, 2Université de Montréal & Mila, 3Pinterest {xq.wang, siliang}@zju.edu.cn bang.liu@umontreal.ca, lwu@email.wm.edu Abstract Existing metrics for assessing question gen- eration not only require costly human refer- ence but also fail to take into account the in- put context of generation, rendering the lack of deep understanding of the relevance be- tween the generated questions and input con- texts. As a result, they may wrongly penal- ize a legitimate and reasonable candidate ques- tion when it (i) involves complicated reason- ing with the context or (ii) can be grounded by multiple evidences in the context. In this pa- per, we propose QRelScore, a context-aware Relevance evaluation metric for Question Gen- eration. Based on off-the-shelf language mod- els such as BERT and GPT2, QRelScore em- ploys both word-level hierarchical matching and sentence-level prompt-based generation to cope with the complicated reasoning and di- verse generation from multiple evidences, re- spectively. Compared with existing metrics, our experiments demonstrate that QRelScore is able to achieve a higher correlation with hu- man judgments while being much more robust to adversarial samples. 1 Introduction Question generation (QG) systems aim to gener- ate natural language questions that are relevant to and usually can be answered by a given piece of input text (Chen et al., 2019c; Liu et al., 2019a, 2020). QG can be used to improve various applica- tions, such as question answering (QA) (Chen et al., 2019a; Fabbri et al., 2020; Yu et al., 2020b; Cheng et al., 2021), conversational systems (Wang et al., 2018; Chen et al., 2019b), and information retrieval (IR) (Yu et al., 2020a; Zamani et al., 2020). Mean- while, it has long been criticized that QG models ∗Equal contribution. †Canada CIFAR AI Chair. ‡Corresponding authors. usually suffer from the semantic drift problem ow- ing to the widely adopted likelihood-based training, i.e. the models ask questions that are not relevant to and can not be supported by the context (Zhang and Bansal, 2019; Chen et al., 2020). Thus, how to accurately evaluate the relevance between gen- erated questions and the context is attracting more and more attention. One of the most accurate eval- uation methods is human evaluation. However, human evaluation is expensive, time-consuming, and non-reproducible. Therefore, it is necessary to develop automatic evaluation metrics for question generation systems. Traditional automatic metrics (e.g. BLEU (Pap- ineni et al., 2002), ROUGE (Lin, 2004) and ME- TEOR (Banerjee and Lavie, 2005)) measure the n-gram overlap between the candidate and corre- sponding reference question, but they often fail to robustly match paraphrases. More recently, Q- BLEU (Nema and Khapra, 2018) and BERT-based metrics such as BERTScore (Zhang et al., 2019), MoverScore (Zhao et al., 2019) and LS_Score (Wu et al., 2020) were proposed to evaluate the answer- ability and semantic similarity of …

Content truncated to 500 words