Dataset Geography: Mapping Language Data to Language Users

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 3381 - 3411 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Dataset Geography: Mapping Language Data to Language Users Fahim Faisal, Yinkai Wang, Antonios Anastasopoulos Department of Computer Science, George Mason University, USA {ffaisal, ywang88, antonis}@gmu.edu Abstract As language technologies become more ubiq- uitous, there are increasing efforts towards ex- panding the language diversity and coverage of natural language processing (NLP) systems. Arguably, the most important factor influenc- ing the quality of modern NLP systems is data availability. In this work, we study the geo- graphical representativeness of NLP datasets, aiming to quantify if and by how much do NLP datasets match the expected needs of the language speakers. In doing so, we use en- tity recognition and linking systems, present- ing an approach for good-enough entity link- ing without entity recognition first. Last, we explore some geographical and economic fac- tors that may explain the observed dataset dis- tributions.1 1 Introduction The lack of linguistic, typological, and geographi- cal diversity in NLP research, authorship, and pub- lications is by now widely acknowledged and doc- umented (Caines, 2019; Ponti et al., 2019; Ben- der, 2011; Adelani et al., 2021). Nevertheless, the advent of massively multilingual models presents opportunity and hope for the millions of speakers of under-represented languages that are currently under-served by language technologies. Broadening up the NLP community’s research efforts and scaling from a handful up to the al- most 7000 languages of the world is no easy feat. In order for this effort to be efficient and success- ful, the community needs some necessary founda- tions to build upon. In seminal work, Joshi et al. (2020) provide a clear overview of where we cur- rently stand with respect to data availability for the world’s languages and relate them to the languages’ representation in NLP conferences. Choudhury and 1Code and data are publicly available: https://github. com/ffaisal93/dataset_geography. Additional visualiza- tions are available in the project page: https://nlp.cs.gmu. edu/project/datasetmaps/. Figure 1: Example of the dataset map our method pro- duces for the Swahili section of MasakhaNER. The dataset is only somewhat representative of Swahili speakers, with only about 17% of entity mentions re- lated to Tanzania, Kenya, Uganda, DR. Congo, or Rwanda and neighboring countries, with the USA and western Europe over-represented. Deshpande (2021) study how linguistically fair are multilingual language models, and provide a nu- anced framework for evaluating multilingual mod- els based on the principles of fairness in economics and social choice theory. Last, Blasi et al. (2022) provide a framework for relating NLP systems’ performance on benchmark datasets to their down- stream utility for users at a global scale, which can provide insights into development priorities; they also discuss academic incentives and socioeco- nomic factors that correlate with the current status of systematic cross-lingual inequalities they ob- serve in language technologies performance. These works provide insights into current data availability and estimated utility that are paramount for making progress, as …

Content truncated to 500 words