Balancing out Bias: Achieving Fairness Through Balanced Training

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 1170 - 1185 December 7-11, 2022 ©2022 Association for Computational Linguistics Multilingual Machine Translation with Hyper-Adapters Christos Baziotis⋆†▽ Mikel Artetxe△ James Cross△ Shruti Bhosale⋆△ ▽ Institute for Language, Cognition and Computation School of Informatics, University of Edinburgh △ Meta AI Research, Menlo Park, CA, USA Abstract Multilingual machine translation suffers from negative interference across languages. A com- mon solution is to relax parameter sharing with language-specific modules like adapters. How- ever, adapters of related languages are unable to transfer information, and their total number of parameters becomes prohibitively expen- sive as the number of languages grows. In this work, we overcome these drawbacks using hyper-adapters—hyper-networks that generate adapters from language and layer embeddings. While past work had poor results when scal- ing hyper-networks, we propose a rescaling fix that significantly improves convergence and enables training larger hyper-networks. We find that hyper-adapters are more parameter efficient than regular adapters, reaching the same performance with up to 12 times less parameters. When using the same number of parameters and FLOPS, our approach con- sistently outperforms regular adapters. Also, hyper-adapters converge faster than alterna- tive approaches and scale better than regular dense networks. Our analysis shows that hyper- adapters learn to encode language relatedness, enabling positive transfer across languages. 1 Introduction Multilingual neural machine translation (MNMT) models (Ha et al., 2016; Johnson et al., 2017) re- duce operational costs and scale to a large number of language pairs (Aharoni et al., 2019) by using a shared representation space. This approach benefits low-resource languages through positive transfer from related languages, but introduces a transfer- interference trade-off (Arivazhagan et al., 2019)— as the number of languages grows, the performance in more resource-rich languages starts to drop. Prior work shows that constrained model capacity prevents models from representing all languages ⋆Correspondence to c.baziotis@ed.ac.uk or shru@fb.com †Work done during an internship at Meta AI Adapter Feed-Forward Self-Attention Transformer Encoder N Adapter Feed-Forward Self-Attention Transformer Decoder N Cross-Attention Hyper-Network zh ko ja bg ru uk en de fr it es dec6 dec3 dec1 enc1 enc6 enc4 enc2 source target layer “[2en] Chinese source sentence” “English target sentence” Figure 1: We inject language(-pair)-specific adapters in MNMT, by generating them from a hyper-network. equally well (Arivazhagan et al., 2019). While naively increasing capacity is certain to improve performance (Arivazhagan et al., 2019; Zhang et al., 2020), it comes with large computational costs. A common remedy for the capacity bottle- neck is to relax the information sharing with lan- guage-specific parameters (Blackwood et al., 2018; Sachan and Neubig, 2018; Wang et al., 2019; Tan et al., 2019; Zhang et al., 2020; Fan et al., 2021). Adapter modules (Rebuffi et al., 2017) have been successfully employed in various natural language processing tasks to address similar capacity-related issues (Houlsby et al., 2019; Pfeiffer et al., 2020). In MNMT, adapters have been used to adapt (via finetuning) pretrained generic models to specific language-pairs or domains(Bapna and Firat, 2019), to improve zero-shot …

Content truncated to 500 words