Systematic Inequalities in Language Technology Performance across the World’s Languages

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 5486 - 5505 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Systematic Inequalities in Language Technology Performance across the World’s Languages Damián Blasi Harvard University dblasi@fas.harvard.edu Antonios Anastasopoulos George Mason University antonis@gmu.edu Graham Neubig Carnegie Mellon University gneubig@cs.cmu.edu Abstract Natural language processing (NLP) systems have become a central technology in commu- nication, education, medicine, artificial intel- ligence, and many other domains of research and development. While the performance of NLP methods has grown enormously over the last decade, this progress has been restricted to a minuscule subset of the world’s ≈6,500 lan- guages. We introduce a framework for estimat- ing the global utility of language technologies as revealed in a comprehensive snapshot of re- cent publications in NLP. Our analyses involve the field at large, but also more in-depth stud- ies on both user-facing technologies (machine translation, language understanding, question answering, text-to-speech synthesis) as well as foundational NLP tasks (dependency parsing, morphological inflection). In the process, we (1) quantify disparities in the current state of NLP research, (2) explore some of its associ- ated societal and academic factors, and (3) pro- duce tailored recommendations for evidence- based policy making aimed at promoting more global and equitable language technologies.1 1 Introduction The past decade has seen a rapid advance in natural language processing (NLP); it has grown from a relatively technical niche to a fundamental tool in virtually all domains that involve language data in any shape or form. NLP is now instrumental to not only bread-and-butter applications such as transla- tion and question answering, but also tasks as wide ranging as detection of neurodegenerative diseases (Orimaye et al., 2017), exposing widespread gen- der and ethnic biases in societies (Caliskan et al., 2017), and predicting large-scale trends in collec- tive consumer behavior (Kallus, 2014). Because of this NLP has become a staple technology for 1Data and code to reproduce the findings discussed in this paper are available on GitHub (https://github.com/ neubig/globalutility). everyday frequent tasks in most contemporary soci- eties of the world. For instance, an English speaker with a smartphone can now easily get accurate in- formation on many topics through a quick query to a virtual assistant, they can consult an online translation service to translate a foreign language web page with a click, and they can interact with many different machines and computers through simple speech commands. These technological capabilities can be at- tributed to several developments over the last few decades: 1. the advent of deep learning methods, which allow for more effective creation of NLP systems from existing data (Goldberg, 2017), 2. the existence of standardized benchmark datasets and evaluation metrics (Wang et al., 2018; Hu et al., 2020), 3. the prestige afforded by the research community to researchers who improve upon these benchmarks, 4. the resulting large number of re- sources, be they computation, data, or ingenu- ity, that are poured into optimizing performance thereon. As both a theoretical …

Content truncated to 500 words