Generating Biographies on Wikipedia: The Impact of Gender Bias on the Retrieval-Based Generation of Women Biographies

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 8561 - 8576 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Generating Full Length Wikipedia Biographies The Impact of Gender Bias on the Retrieval-Based Generation of Women Biographies Angela Fan FAIR / LORIA Université de Lorraine angelafan@fb.com Claire Gardent CNRS/LORIA Nancy, France claire.gardent@loria.fr Abstract Generating factual, long-form text such as Wikipedia articles raises three key challenges: how to gather relevant evidence, how to struc- ture information into well-formed text, and how to ensure that the generated text is factually cor- rect. We address these by developing a model for English text that uses a retrieval mechanism to identify relevant supporting information on the web and a cache-based pre-trained encoder- decoder to generate long-form biographies sec- tion by section, including citation information. To assess the impact of available web evidence on the output text, we compare the performance of our approach when generating biographies about women (for which less information is available on the web) vs. biographies generally. To this end, we curate a dataset of 1,500 biogra- phies about women. We analyze our generated text to understand how differences in available web evidence data affect generation. We eval- uate the factuality, fluency, and quality of the generated texts using automatic metrics and hu- man evaluation. We hope that these techniques can be used as a starting point for human writ- ers, to aid in reducing the complexity inherent in the creation of long-form, factual text. 1 Introduction Wikipedia has become one of the major sources of dissemination of knowledge across the globe. How- ever, the knowledge contained in Wikipedia is not neutral — it is biased in various ways (Hinnosaar, 2019; Schmahl et al., 2020). Many studies, includ- ing those from the Wikimedia Foundation itself, have emphasized that biographies in particular are overwhelmingly written about men. This leads to many subtle yet far-reaching effects, from students not writing their first book reports on a woman to bias in models trained on Wikipedia, as Wikipedia has long been used as a source of data. Many ex- isting efforts, such as the Wikipedia Women in Red project, focus on encouraging article creation to mitigate this gender gap. However, Wikipedia articles remain painstakingly written and edited primarily by a network of human contributors. De- spite advances in text generation and modeling ar- chitectures that retrieve information, the automatic creation of Wikipedia articles is incredibly chal- lenging (Liu et al., 2018). Even the functionality of tools that aid human editors are limited. In this work, we strive to create a system that could write an entire Wikipedia article in English, focusing on the biography domain. We confront several major challenges. First, this is funda- mentally a long-form generation task. Improve- ments driven by pretraining (Radford et al., 2019; Lewis et al., 2019) have improved generation flu- ency at the level of multiple sentences. However, Wikipedia biographies contain multiple paragraphs in a structured form with headings, as well …

Content truncated to 500 words