Beyond Metadata: What Paper Authors Say About Corpora They Use

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 5085–5090 August 1–6, 2021. ©2021 Association for Computational Linguistics 5085 Beyond Metadata: What Paper Authors Say About Corpora They Use Nikolay Kolyada1 Martin Potthast2 Benno Stein1 1 Bauhaus-Universität Weimar, Germany, <first>.<last>@uni-weimar.de 2 Leipzig University, Germany, martin.potthast@uni-leipzig.de Abstract The growing ecosystem of data sharing in sci- ence has put dataset search into the focus. To make data sharing and reuse more feasible, new retrieval tools and services are being de- veloped. Currently, dataset retrieval relies al- most exclusively on metadata provided by the publishers. To extend this knowledge source our work studies the task of “dataset review mining” in scientific publications. For the field of Natural Language Processing we col- lect metadata about datasets from established resources such as the ELRA and LDC catalogs, and then extract review statements about the datasets from ACL Anthology Corpus publica- tions, compiling the Webis-Dataset-Reviews- 21 corpus. By analyzing the reviews we iden- tify different categories of what paper authors write about data. To the best of our knowledge, this is the first analysis of this kind in the field of Natural Language Processing, albeit similar analyses have been carried out in the social and medical sciences. Our corpus and the underly- ing code are shared alongside this paper.1,2,3 1 Introduction Recently, Google introduced its Dataset Search ser- vice (Brickley et al., 2019).4 Around 30 million datasets have been indexed to date,5 hosted at web pages and repositories all across the web. Although data sharing and dataset search has long been a best practice in natural language processing, Google’s service has accelerated and standardized data shar- ing across scientific communities, since only those datasets are indexed, for which a certain metadata description prescribed by Google is provided. 1https://github.com/webis-de/ACL-21 2https://webis.de/data.html#Webis-Dataset-Reviews-21 3https://doi.org/10.5281/zenodo.4889032 4https://datasetsearch.research.google.com/ 5https://ai.googleblog.com/2020/08/an-analysis-of-online- datasets-using.html Chapman et al. (2020) and Koesten et al. (2017) survey the state of the art in dataset search and analyze how it differs from classical information retrieval. Approaches to dataset search from re- lated fields, such as databases, semantic web, entity- based search, and tabular search are reviewed, but neither solves the task comprehensively. From a user’s perspective, searching for datasets entails two steps: (1) its retrieval using standard keyword search, and (2) examining the search results to iden- tify the datasets suitable to one’s needs. The sec- ond step strongly differs from web search, since a (large) dataset cannot be easily reviewed in the browser; merely it’s metadata is available—as well as the descriptions found on the download page. We argue that a crucial piece of information is missing to inform the examination of dataset search results: dataset user experience. Unlike for commercial products in online shops, for datasets, there are hardly any platforms where users share their experiences using them. On the few that do exist, the amount of published reviews cannot be compared with that of commercial prod- ucts. In any case, due to the specificity of most datasets, few people actually use them. Their feed- back is typically …

Content truncated to 500 words