Bayes Test of Precision, Recall, and F1 Measure for Comparison of Two Natural Language Processing Models

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4135–4145 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 4135 Bayes Test of Precision, Recall, and F1 Measure for Comparison of Two Natural Language Processing Models Ruibo Wang School of Modern Education Technology Shanxi University Taiyuan, China, 030006 wangruibo@sxu.edu.cn Jihong Li School of Software Shanxi University Taiyuan, China, 030006 li ml@sxu.edu.cn Abstract Direct comparison on point estimation of the precision (P), recall (R), and F1 measure of t- wo natural language processing (NLP) models on a common test corpus is unreasonable and results in less replicable conclusions due to a lack of a statistical test. However, the exist- ing t-tests in cross-validation (CV) for model comparison are inappropriate because the dis- tributions of P, R, F1 are skewed and an inter- val estimation of P, R, and F1 based on a t-test may exceed [0,1]. In this study, we propose to use a block-regularized 3 × 2 CV (3 × 2 BCV) in model comparison because it could regular- ize the difference in certain frequency distri- butions over linguistic units between training and validation sets and yield stable estimators of P, R, and F1. On the basis of the 3 × 2 BCV, we calibrate the posterior distributions of P, R, and F1 and derive an accurate inter- val estimation of P, R, and F1. Furthermore, we formulate the comparison into a hypothe- sis testing problem and propose a novel Bayes test. The test could directly compute the prob- abilities of the hypotheses on the basis of the posterior distributions and provide more infor- mative decisions than the existing significance t-tests. Three experiments with regard to NLP chunking tasks are conducted, and the results illustrate the validity of the Bayes test. 1 Introduction The comparison of two models is a key step in natural language processing (NLP) with the preci- sion (P), recall (R), and F1 measures. The com- parison could be described as follows: For two NLP models on a given text corpus, which model produces a higher performance system with a rel- atively high probability? The direct comparison with a point estimation of P, R, and F1 on a test corpus is unscientific from a statistical perspective and usually leads to less replicable results (Dror et al., 2017). In reality, the comparison generally could be formalized with a statistical hypothesis testing, and many prominent tests, such as K-fold cross-validated (CV) t-test (Daelemans and Hoste, 2002), 5×2 CV t-test and F-test (Dietterich, 1998; Alpaydin, 1999), and block-regularized 3 × 2 CV (3 × 2 BCV) t-test (Wang et al., 2014), have been conducted. However, the distributions of P, R, and F1 are skewed (Wang et al., 2015) and take values in [0, 1], but an interval estimation of P, R, and F1 based on a t-test may exceed [0,1]. In this study, we introduce a Bayes test that is more informative than the previous prominent nul- l hypothesis significance testing (NHST) …

Content truncated to 500 words