infoVerse: A Universal Framework for Dataset Characterization with Multidimensional Meta-information

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 9823–9850 July 9-14, 2023 ©2023 Association for Computational Linguistics infoVerse: A Universal Framework for Dataset Characterization with Multidimensional Meta-information Jaehyung Kim†∗ Yekyung Kim‡ Karin de Langis♢ Jinwoo Shin† Dongyeop Kang♢ †KAIST, ‡Hyundai Motors Company, ♢University of Minnesota jaehyungkim@kaist.ac.kr dongyeop@umn.edu Abstract The success of NLP systems often relies on the availability of large, high-quality datasets. However, not all samples in these datasets are equally valuable for learning, as some may be redundant or noisy. Several methods for characterizing datasets based on model- driven meta-information (e.g., model’s confi- dence) have been developed, but the relation- ship and complementary effects of these meth- ods have received less attention. In this paper, we introduce infoVerse, a universal framework for dataset characterization, which provides a new feature space that effectively captures multidimensional characteristics of datasets by incorporating various model-driven meta- information. infoVerse reveals distinctive re- gions of the dataset that are not apparent in the original semantic space, hence guiding users (or models) in identifying which samples to focus on for exploration, assessment, or anno- tation. Additionally, we propose a novel sam- pling method on infoVerse to select a set of data points that maximizes informativeness. In three real-world applications (data pruning, ac- tive learning, and data annotation), the samples chosen on infoVerse space consistently outper- form strong baselines in all applications. Our code and demo are publicly available.1 1 Introduction The construction of large datasets is one of the es- sential ingredients for success in various NLP tasks (Wang et al., 2019). However, not all data points are equally important to learn from; many datasets often contain low-quality samples, e.g., incorrect labels (Toneva et al., 2019) or annotation artifacts (Gururangan et al., 2018). Thus, data character- ization (Roth and Mattis, 1990), a technique for transforming raw data into useful information for ∗This work was done while JK was at Minnesota NLP. 1https://github.com/minnesotanlp/ infoVerse Data Pruning Active Learning Data Annotation Data-centric Applications Training Dynamics Model Uncertainty Pre-trained Knowledge Static Measures Model-driven Meta-information Maximally Informative Subset Selection Data Points infoVerse Hard-to-learn Noisy Ambiguous Out-of-domain 2 1 3 4 5 1 2 3 4 5 Figure 1: The proposed framework, infoVerse. By incorporating multidimensional aspects of data charac- teristics, infoVerse enables a better dataset characteri- zation. By selecting maximally informative subsets on infoVerse, we improve model performance on a variety of data-centric real-world problems like active learning. a target task, has a huge potential to improve the model’s performance by trimming the problematic samples (Pleiss et al., 2020) or providing better practices for effective data collection, e.g., active learning (Beluch et al., 2018) and adversarial anno- tation (Nie et al., 2019). However, data characterization via human as- sessment is highly limited due to the huge cost of dealing with a large dataset and the vagueness of the assessment itself. To this end, several model- 9823 driven meta-information2 have been investigated; for example, the model’s confidence is a standard meta-information widely …

Content truncated to 500 words