Comparing Intrinsic Gender Bias Evaluation Measures without using Human Annotated Examples
42841Authors
nan
Abstract
AbstractNumerous types of social biases have been identified in pre-trained language models (PLMs), and various intrinsic bias evaluation measures have been proposed for quantifying those social biases. Prior works have relied on human annotated examples to compare existing intrinsic bias evaluation measures. However, this approach is not easily adaptable to different languages nor amenable to large scale evaluations due to the costs and difficulties when recruiting human annotators. To overcome this limitation, we propose a method to compare intrinsic gender bias evaluation measures without relying on human-annotated examples. Specifically, we create multiple bias-controlled versions of PLMs using varying amounts of male vs. female gendered sentences, mined automatically from an unannotated corpus using gender-related word lists. Next, each bias-controlled PLM is evaluated using an intrinsic bias evaluation measure, and the rank correlation between the computed bias scores and the gender proportions used to fine-tune the PLMs is computed. Experiments on multiple corpora and PLMs repeatedly show that the correlations reported by our proposed method that does not require human annotated examples are comparable to those computed using human annotated examples in prior work.
Keywords
Full Content
Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 265–273 May 2-6, 2023 ©2023 Association for Computational Linguistics A Discerning Several Thousand Judgments: GPT-3 Rates the Article + Adjective + Numeral + Noun Construction Kyle Mahowald mahowald@utexas.edu Department of Linguistics The University of Texas at Austin Abstract Knowledge of syntax includes knowledge of rare, idiosyncratic constructions. LLMs must overcome frequency biases in order to mas- ter such constructions. In this study, I prompt GPT-3 to give acceptability judgments on the English-language Article + Adjective + Nu- meral + Noun construction (e.g., “a lovely five days”). I validate the prompt using the CoLA corpus of acceptability judgments and then zero in on the AANN construction. I compare GPT- 3’s judgments to crowdsourced human judg- ments on a subset of sentences. GPT-3’s judg- ments are broadly similar to human judgments and generally align with proposed constraints in the literature but, in some cases, GPT-3’s judgments and human judgments diverge from the literature and from each other. 1 Introduction Consider the English Article + Adjective + Nu- meral + Noun (AANN) construction: “ a beautiful 228 pages [iWeb]” or “The president has had a ter- rible five weeks [COCA]”. Usually cardinal num- bers precede the adjective (“five terrible weeks”), but here the adjective precedes the numeral. More strangely, the normally singular article “a” in this case is followed by a plural noun phrase.1 An eclectic dozen or so papers have been writ- ten on the construction, many focused on eluci- dating relevant semantic and syntactic constraints (Goldberg and Michaelis, 2017; Jackendoff, 1977; Dalrymple and King, 2019; Bylinina and Nouwen, 2018; Ionin and Matushansky, 2018, 2004; Solt, 2007; Keenan, 2013). The presence of the modifier is crucial: “a 228 pages” is unacceptable. The type of modifier is also crucial: “a pink 228 pages” is odd because color words are “stubbornly distribu- tive” (Schwarzschild, 2011) and thus cannot refer to a set of items as a whole: the nominal phrase 1Data and code: https://github.com/mahowak/ aann-public/ Figure 1: GPT-3 acceptability judgments (bars), com- pared to human ratings (green triangles) on a matched set of sentences. The comparison is between the AANN construction and the standard alternative, as well as 4 degenerate versions. Both humans and GPT-3 rank the AANN construction as being as acceptable as the standard, and all degenerate constructions are rated sig- nificantly lower. needs to function as a unit (Solt, 2007). These id- iosyncratic constraints are typical of constructions (Goldberg, 2019). Prior work on the AANN construction has fo- cused on characterizing the semantic and syntac- tic constraints on the construction and proposing analyses in various frameworks. For instance, Solt (2007) focuses on how the construction coerces the phrase into a singular noun phrase; Keenan (2013) proposes treating it as akin to a partitive, and Dal- rymple and King (2019) give an LFG analysis. The same properties that make AANN inter- esting from the perspective of human language use—its low frequency but high sensitivity to constraints—also …
Content truncated to 500 words
Labeled Text
AbstractNumerous types of social biases have been identified in pre-trained language models (PLMs), and various intrinsic bias evaluation measures have been proposed for quantifying those social biases. Prior works have relied on human annotated examples to compare existing intrinsic bias evaluation measures. However, this approach is not easily adaptable to different languages nor amenable to large scale evaluations due to the costs and difficulties when recruiting human annotators. To overcome this limitation, we propose a method to compare intrinsic gender bias evaluation measures without relying on human-annotated examples. Specifically, we create multiple bias-controlled versions of PLMs using varying amounts of male vs. female gendered sentences, mined automatically from an unannotated corpus using gender-related word lists. Next, each bias-controlled PLM is evaluated using an intrinsic bias evaluation measure, and the rank correlation between the computed bias scores and the gender proportions used to fine-tune the PLMs is computed. Experiments on multiple corpora and PLMs repeatedly show that the correlations reported by our proposed method that does not require human annotated examples are comparable to those computed using human annotated examples in prior work.
Publication Details
- Paper ID:
- 42841
- Year:
- 2023
- Publication:
- TIPSTER
- Publication Type:
- conf
- DOI:
- nan