DataFinder: Scientific Dataset Recommendation from Natural Language Descriptions

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 10288–10303 July 9-14, 2023 ©2023 Association for Computational Linguistics DataFinder: Scientific Dataset Recommendation from Natural Language Descriptions Vijay Viswanathan1 Luyu Gao1 Tongshuang Wu1 Pengfei Liu2,3 Graham Neubig1,3 1Carnegie Mellon University 2Shanghai Jiao Tong University 3Inspired Cognition {vijayv, luyug, sherryw, gneubig}@cs.cmu.edu stefanpengfei@gmail.com Abstract Modern machine learning relies on datasets to develop and validate research ideas. Given the growth of publicly available data, finding the right dataset to use is increasingly difficult. Any research question imposes explicit and implicit constraints on how well a given dataset will en- able researchers to answer this question, such as dataset size, modality, and domain. We op- erationalize the task of recommending datasets given a short natural language description of a research idea, to help people find relevant datasets for their needs. Dataset recommenda- tion poses unique challenges as an information retrieval problem; datasets are hard to directly index for search and there are no corpora read- ily available for this task. To facilitate this task, we build the DataFinder Dataset which con- sists of a larger automatically-constructed train- ing set (17.5K queries) and a smaller expert- annotated evaluation set (392 queries). Using this data, we compare various information re- trieval algorithms on our test set and present a superior bi-encoder retriever for text-based dataset recommendation. This system, trained on the DataFinder Dataset, finds more relevant search results than existing third-party dataset search engines. To encourage progress on dataset recommendation, we release our dataset and models to the public.1 1 Introduction Innovation in modern machine learning (ML) de- pends on datasets. The revolution of neural net- work models in computer vision (Krizhevsky et al., 2012) was enabled by the ImageNet Large Scale Visual Recognition Challenge (Deng et al., 2009). Similarly, data-driven models for syntactic pars- ing saw rapid development after adopting the Penn Treebank (Marcus et al., 1993; Palmer and Xue, 2010). 1Code and data: https://github.com/viswavi/ datafinder Natural Language Query “I want to use adversarial learning to perform domain adaptation for semantic segmentation of images.” Keyword Query “semantic segmentation domain adaptation images” GTA5 ADE20K Cityscape Query Interface (Explicit) - Image semantic segmentation (Implicit) - Image semantic segmentation - Datasets should include diverse domains. Relevant datasets Constraints (task/modality/etc) fef7e0 F8F9FA FEEFE3 fad2cf GTA5 ADE20K PACS Cityscape SYNTHIA Figure 1: Queries for dataset recommendation im- pose constraints on the type of dataset desired. Key- word queries make these constraints explicit, while full- sentence queries impose implicit constraints. Ground truth relevant datasets for this query are colored in blue. With the growth of research in ML and artificial intelligence (AI), there are hundreds of datasets published every year (shown in Figure 2). Knowing which to use for a given research idea can be diffi- cult (Paullada et al., 2021). To illustrate, consider a real query from a graduate student who says, “I want to use adversarial learning to perform domain adaptation for semantic segmentation of images.” They have implicitly issued two requirements: …

Content truncated to 500 words