KPQA: A Metric for Generative Question Answering Using Keyphrase Weights

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2105–2115 June 6–11, 2021. ©2021 Association for Computational Linguistics 2105 KPQA: A Metric for Generative Question Answering Using Keyphrase Weights Hwanhee Lee1, Seunghyun Yoon2, Franck Dernoncourt2 Doo Soon Kim3∗, Trung Bui2, Joongbo Shin1 and Kyomin Jung1 1Dept. of Electrical and Computer Engineering, Seoul National University, Seoul, Korea 2Adobe Research, San Jose, CA, USA, 3Roku Inc., San Jose, CA, USA {wanted1007, jbshin, kjung}@snu.ac.kr {syoon, franck.dernoncourt, bui}@adobe.com, dskim@roku.com Abstract In the automatic evaluation of generative ques- tion answering (GenQA) systems, it is diffi- cult to assess the correctness of generated an- swers due to the free-form of the answer. Espe- cially, widely used n-gram similarity metrics often fail to discriminate the incorrect answers since they equally consider all of the tokens. To alleviate this problem, we propose KPQA- metric, a new metric for evaluating the cor- rectness of GenQA. Specifically, our new met- ric assigns different weights to each token via keyphrase prediction, thereby judging whether a generated answer sentence captures the key meaning of the reference answer. To evaluate our metric, we create high-quality human judg- ments of correctness on two GenQA datasets. Using our human-evaluation datasets, we show that our proposed metric has a significantly higher correlation with human judgments than existing metrics. Code for KPQA-metric will be available at https://github.com/ hwanheelee1993/KPQA. 1 Introduction Question answering (QA) has received consistent attention from the natural language processing com- munity. Recently, research on QA systems has reached the stage of generating free-form answers, called GenQA, beyond extracting the answer to a given question from the context (Yin et al., 2016; Song et al., 2017; Bauer et al., 2018; Nishida et al., 2019; Bi et al., 2019, 2020). However, as a bot- tleneck in developing GenQA models, there are no proper automatic metrics to evaluate generated answers (Chen et al., 2019). In evaluating a GenQA model, it is essential to consider whether a generated response correctly contains vital information to answer the question. There exist several n-gram similarity metrics such ∗This research was done while the author was affiliated with Adobe Research. Context : ... , this process, called hypothesis testing, consists of four steps. , ... Question : How many steps are involved in a hy- pothesis test? Reference Answer : Four steps are involved in a hypothesis test. Generated Answer : There are seven steps involved in a hypothesis test . Human Judgment : 0.063 BLEU-1 : 0.778 BLEU-1-KPQA : 0.057 ROUGE-L : 0.713 ROUGE-L-KPQA : 0.127 Figure 1: An example from MS-MARCO (Bajaj et al., 2016) where widely used n-gram similarity metrics does not align with human judgments of correctness. On the other hand, our KPQA-metrics focus on the key information and give low scores to incorrect answers similar to humans. as BLEU (Papineni et al., 2002) and ROUGE- L (Lin, 2004), that measure the word overlaps be- tween the generated response and the reference answer; however, these metrics are …

Content truncated to 500 words