GeBioToolkit: Automatic Extraction of Gender-Balanced Multilingual Corpus of Wikipedia Biographies

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 4081–4088 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 4081 GeBioToolkit: Automatic Extraction of Gender-Balanced Multilingual Corpus of Wikipedia Biographies Marta R. Costa-juss`a, Pau Li Lin, Cristina Espa˜na-Bonet∗ TALP Research Center, Universitat Polit`ecnica de Catalunya, Barcelona ∗ DFKI GmBH and Saarland University, Saarbr¨ucken marta.ruiz@upc.edu, lilin.pau@gmail.com, cristinae@dfki.de Abstract We introduce GeBioToolkit, a tool for extracting multilingual parallel corpora at sentence level, with document and gender information from Wikipedia biographies. Despite the gender inequalities present in Wikipedia, the toolkit has been designed to extract corpus balanced in gender. While our toolkit is customizable to any number of languages (and to other domains than biographical entries), in this work we present a corpus of 2,000 sentences in English, Spanish and Catalan, which has been post-edited by native speakers to become a high-quality dataset for machine translation evaluation. While GeBioCorpus aims at being one of the first non-synthetic gender-balanced test datasets, GeBioToolkit aims at paving the path to standardize procedures to produce gender-balanced datasets. Keywords: corpora, gender bias, Wikipedia, machine translation 1. Introduction Gender biases are present in many natural language pro- cessing applications (Costa-juss`a, 2019). This comes as an undesired characteristic of deep learning architectures where their outputs seem to reflect demographic asymme- tries (Prates et al., 2019). This is of course not due to the architecture itself but to the data used to train a system. Re- cent research is being devoted to correct the asymmetries mainly by data augmentation techniques in fields such as coreference resolution (Rudinger et al., 2018; Zhao et al., 2018; Webster et al., 2018) or abusive language detection (Park et al., 2018). Test sets have been created in those cases, but we are not aware of any test set available for ma- chine translation (MT). From another side, machine translation either neural, sta- tistical or rule-based, usually operates in a sentence-by- sentence basis. However, when translating consistently a document, surrounding sentences may have valuable infor- mation. The translation of pronouns, verb tenses and even content words might depend on other fragments within a document. This affects also the translation of the gender markers, specially when translating from a language with- out these marks (e.g. English) into a language with them (e.g. Catalan). Test sets at sentence level are not useful to evaluate these phenomena. But in the time of claims of human parity in MT, all these phenomena are crucial, and document-level evaluation is needed in order to discrimi- nate among systems (L¨aubli et al., 2018). Beyond these gender-balanced and document-level needs, the rise of multilingual neural machine translation and the lack of multi-way parallel corpus that can evaluate its abil- ities (e.g. zero-shot), motivates the creation of new multi- lingual corpora. In turn, this motivation prompts the devel- opment of software to automatically create such corpora. In order to create a GeBioToolkit that is able to system- atically extract multilingual parallel corpus at sentence level and with document-level …

Content truncated to 500 words