On Evaluation of Document Classification with RVL-CDIP

Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 2665–2678 May 2-6, 2023 ©2023 Association for Computational Linguistics On Evaluation of Document Classification with RVL-CDIP Stefan Larson1⇤ Gordon Lim2 1Vanderbilt University 2University of Michigan Nashville, TN, USA Ann Arbor, MI, USA Kevin Leach1 Abstract The RVL-CDIP benchmark is widely used for measuring performance on the task of docu- ment classification. Despite its widespread use, we reveal several undesirable characteristics of the RVL-CDIP benchmark. These include (1) substantial amounts of label noise, which we estimate to be 8.1% (ranging between 1.6% to 16.9% per document category); (2) pres- ence of many ambiguous or multi-label doc- uments; (3) a large overlap between test and train splits, which can inflate model perfor- mance metrics; and (4) presence of sensitive personally-identifiable information like US So- cial Security numbers (SSNs). We argue that there is a risk in using RVL-CDIP for bench- marking document classifiers, as its limited scope, presence of errors (state-of-the-art mod- els now achieve accuracy error rates that are within our estimated label error rate), and lack of diversity make it less than ideal for bench- marking. We further advocate for the creation of a new document classification benchmark, and provide recommendations for what charac- teristics such a resource should include. 1 Introduction Within the document understanding research area, the RVL-CDIP dataset (Harley et al., 2015) has emerged as the primary benchmark for evaluat- ing and comparing document classifiers. RVL- CDIP is composed of 16 document type categories, including resume, letter, invoice, etc. Its large volume of training data—320,000 samples— facilitates benchmarking state-of-the-art deep learn- ing and transformer-based architectures. While ini- tially released as a computer vision benchmark in 2015, more recent state-of-the-art models now in- corporate image, text, and page layout modalities. For instance, recent tri-modal models like Doc- Former (Appalaraju et al., 2021), ERNIE-Layout (Peng et al., 2022), LayoutLMv3 (Huang et al., ⇤ Corresponding email: stefan.dataset@gmail.com Figure 1: Model accuracy on RVL-CDIP by year and modality. The horizontal dashed line represents our estimated label error rate for RVL-CDIP’s test set. 2022), and Bi-VLDoc (Luo et al., 2022) now achieve classification accuracies ranging in the mid- to high-90s, with Bi-VLDoc reporting a state-of- the-art of 97.12% on the RVL-CDIP test set. This is a large improvement over earlier image-centric work, and we chart this improvement in Figure 1. As model performance on RVL-CDIP improves, it becomes increasingly important to ensure that further gains are meaningful with respect to the classification task. This concern has been raised by prior work that has found that benchmark evalua- tion datasets often contain substantial amounts of label errors or noise (e.g., Northcutt et al., 2021a), substantial overlap between test and train data (e.g., Elangovan et al., 2021; Søgaard et al., 2021), and data collection artifacts that cause models to over- fit to spurious cues (e.g., Gururangan et al., 2018; McCoy et al., 2019). Therefore, we cast a critical eye to the RVL-CDIP benchmark to answer: Is RVL-CDIP still …

Content truncated to 500 words