Evaluating the Diversity, Equity, and Inclusion of NLP Technology: A Case Study for Indian Languages

Findings of the Association for Computational Linguistics: EACL 2023, pages 1763–1777 May 2-6, 2023 ©2023 Association for Computational Linguistics Evaluating the Diversity, Equity and Inclusion of NLP Technology: A Case Study for Indian Languages Simran Khanuja∗ Carnegie Mellon University† skhanuja@andrew.cmu.edu Sebastian Ruder∗ Google Research ruder@google.com Partha Talukdar Google Research partha@google.com Abstract In order for NLP technology to be widely ap- plicable, fair, and useful, it needs to serve a diverse set of speakers across the world’s lan- guages, be equitable, i.e., not unduly biased towards any particular language, and be inclu- sive of all users, particularly in low-resource settings where compute constraints are com- mon. In this paper, we propose an evalua- tion paradigm that assesses NLP technologies across all three dimensions. While diversity and inclusion have received attention in recent literature, equity is currently unexplored. We propose to address this gap using the Gini co- efficient, a well-established metric used for es- timating societal wealth inequality. Using our paradigm, we highlight the distressed state of current technologies for Indian (IN) languages (a linguistically large and diverse set, with a varied speaker population), across all three di- mensions. To improve upon these metrics, we demonstrate the importance of region-specific choices in model building and dataset creation, and more importantly, propose a novel, gen- eralisable approach to optimal resource allo- cation during fine-tuning. Finally, we discuss steps to mitigate these biases and encourage the community to employ multi-faceted evalu- ation when building linguistically diverse and equitable technologies. 1 Introduction NLP has seen large advances in recent years driven by the rapid progress in transfer learning (Ruder et al., 2019; Devlin et al., 2019). The benefits of these advances, however, are not equally dis- tributed across the world’s languages (Joshi et al., 2020) and users. While linguistic diversity and inclusion have evolved to be a pressing concern today, measures to quantify these are still lacking. The progress of any field is tightly coupled with ∗Equal contribution. †Work done at Google Research. its evaluation paradigm and the community is in- centivized to work on highly visible metrics and benchmarks. In order for users around the world to reap the benefits of NLP technology, we must move from an evaluation that focuses on optimiz- ing raw performance on available test data to a more holistic user-centric evaluation (Ethayarajh and Jurafsky, 2020; Ruder et al., 2021). In this paper, we attempt to do so by defining an evalua- tion framework along three dimensions: diversity, equity, and inclusion.1 Diversity is important as NLP technology should be available to speakers of any language (European Language Resources Association, 2019). To this end, recent work (Blasi et al., 2022) quantifies di- versity of NLP technology across the world’s lan- guages by weighing normalized task performance for each language based on its speaker population. Equity is key as we should aim to develop tech- nology that does not discriminate against speakers of any particular language (Kaneko and Bollegala, 2019). State-of-the-art multilingual models in fact have been shown to perform much …

Content truncated to 500 words