Adaptations of ROUGE and BLEU to Better Evaluate Machine Reading Comprehension Task

Proceedings of the Workshop on Machine Reading for Question Answering, pages 98–104 Melbourne, Australia, July 19, 2018. c⃝2018 Association for Computational Linguistics 98 Adaptations of ROUGE and BLEU to Better Evaluate Machine Reading Comprehension Task An Yang1 2*, Kai Liu2, Jing Liu2, Yajuan Lyu2 and Sujian Li1 1Key Laboratory of Computational Linguistics, Peking University, MOE, China 2Baidu Inc., Beijing, China {yangan, lisujian}@pku.edu.cn {liukai20, liujing46, lvyajuan}@baidu.com Abstract Current evaluation metrics to question an- swering based machine reading compre- hension (MRC) systems generally focus on the lexical overlap between candidate and reference answers, such as ROUGE and BLEU. However, bias may appear when these metrics are used for specific question types, especially questions in- quiring yes-no opinions and entity lists. In this paper, we make adaptations on the metrics to better correlate n-gram overlap with the human judgment for answers to these two question types. Statistical anal- ysis proves the effectiveness of our ap- proach. Our adaptations may provide pos- itive guidance for the development of real- scene MRC systems. 1 Introduction The goal of current MRC tasks is to develop agents which are able to comprehend passages automati- cally and answer open-domain questions correctly. With the release of several large-scale datasets like SQuAD (Rajpurkar et al., 2016), MS-MARCO (Nguyen et al., 2016) and DuReader (He et al., 2017), many MRC models have been proposed in previous works (Wang and Jiang, 2016; Seo et al., 2016; Wang et al., 2017). Although MRC model architectures have been intensively studied, the evaluation metrics for them are rarely discussed. For early cloze-style and multiple choice datasets (Richardson et al., 2013; Hermann et al., 2015), this may not be problematic. However, consider- ing the trend that the model is required to generate answers and question type is becoming more vari- able and closer to real cases, we believe the design *This work was done while the first author was doing in- ternship at Baidu Inc. of evaluation metric is indeed an issue to be fo- cused on. Currently, the criterion for comparing generated and gold answers is mostly based on lexical over- lap. For example, SQuAD uses exact-match ra- tio and word-level F1-score, while MS-MARCO and DuReader employ ROUGE-L (Lin, 2004) and BLEU (Papineni et al., 2002) which measure n- gram consistency or longest common sequence (LCS) length. For some question types, we no- tice these metrics may not correlate with seman- tic correspondence well in some cases. In this paper, we mainly tackle the issue of yes-no and entity questions. For yes-no questions, overlap- based metrics may ignore the yes-or-no opinion which is more crucial in determining agreement between answers. Answers with contrary opinions may have high lexical overlap, such as “The radi- ation of wireless routers has an impact on peo- ple” and “The radiation of wireless routers has no impact on people”. Similarly, for entity ques- tions, we think the agreement should be more re- flected by the correctness of entity listing. An- swers which lack or mispredict entities should be in distinction …

Content truncated to 500 words