WebCrawl African : A Multilingual Parallel Corpora for African Languages

Proceedings of the Seventh Conference on Machine Translation (WMT), pages 1076 - 1089 December 7-8, 2022 ©2022 Association for Computational Linguistics WebCrawl African : A Multilingual Parallel Corpora for African Languages Pavanpankaj Vegi, Sivabhavani J, Biswajit Paul, Abhinav Mishra, Prashant Banjare, Prasanna Kumar KR, Chitra Viswanathan pavanpankaj333@gmail.com, {jsbhavani, biswajit, abhinavmishra}.cair@gov.in, {prashantban408, prasanna, chitrav}.cair@gov.in Centre for Artificial Intelligence and Robotics, CV Raman Nagar, Bangalore Abstract WebCrawl African is a mixed domain multilin- gual parallel corpora for a pool of African lan- guages compiled by ANVITA machine transla- tion team of Centre for Artificial Intelligence and Robotics Lab, primarily for accelerating research on low-resource and extremely low- resource machine translation and is part of the submission to WMT 2022 shared task on Large-Scale Machine Translation Evalua- tion for African Languages under the data track. The corpora is compiled through web data mining and comprises 695K parallel sen- tences spanning 74 different language pairs from English and 15 African languages, many of which fall under low and extremely low re- source categories. As a measure of corpora usefulness, a MNMT model for 24 African languages to English is trained by combining WebCrawl African corpora with existing cor- pus and evaluation on FLORES200 shows that inclusion of WebCrawl African corpora could improve BLEU score by 0.01-1.66 for 12 out of 15 African→English translation directions and even by 0.18-0.68 for the 4 out of 9 African→English translation directions which are not part of WebCrawl African corpora. We- bCrawl African corpora includes more par- allel sentences for many language pairs in comparison to OPUS public repository. This data description paper captures creation of cor- pora and results obtained along with datasheet. The WebCrawl African corpora is hosted on GitHub repository 1. 1 Introduction Parallel corpus play a vital role in the progress of data driven machine translation research and devel- opment. Availability of parallel corpora is still a concern for a large collection of world languages. Africa alone is home to an estimated 1200 to 2100 spoken languages2 and more than 34 languages 1https://github.com/pavanpankaj/Web-Crawl-African 2https://en.wikipedia.org/wiki/Languages_of_Africa with 1 Million plus speakers. Many of these 34 languages and associated language pairs fall un- der the low and extremely low resource categories and machine translation researchers face setbacks due to unavailability of parallel corpus in public domain. WebCrawl African corpora is a little step put forward towards addressing this issue. Languages covered in WebCrawl African corpora in- clude (1) Afrikaans(afr), (2) Amharic(amh), (3) Chichewa(nya), (4) Hausa(hau), (5) Igbo(ibo), (6) Lingala(lin), (7) Luganda(lug), (8) Oromo(orm), (9) Swahili(swh), (10) Swati(ssw), (11) Tswana/Setswana(tsn), (12) Xhosa(xho), (13) Xitsonga(tso), (14) Yoruba(yor) (15) Zulu(zul) and (16) English and language pairs include African- English and African-African pairs. WebCrawl African is submitted as a part of Large-Scale Machine Translation Evaluation for African Languages shared task(data track) of WMT22 Adelani et al. (2022). Rest of the paper is organized as follows. Section-2 briefly covers related work on paral- lel corpora compilation through web data mining. Section-3 covers content collection process fol- lowed for WebCrawl African corpora compilation, Section-4 …

Content truncated to 500 words