Phone-ing it in: Towards Flexible Multi-Modal Language Model Training by Phonetic Representations of Data

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 5306 - 5315 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Phone-ing it in: Towards Flexible, Multi-Modal Language Model Training using Phonetic Representations of Data Colin Leong University of Dayton cleong1@udayton.edu Daniel Whitenack SIL International dan_whitenack@sil.org Abstract Multi-modal techniques offer significant un- tapped potential to unlock improved NLP tech- nology for local languages. However, many advances in language model pre-training are focused on text, a fact that only increases sys- tematic inequalities in the performance of NLP tasks across the world’s languages. In this work, we propose a multi-modal approach to train lan- guage models using whatever text and/or audio data might be available in a language. Initial experiments using Swahili and Kinyarwanda data suggest the viability of the approach for downstream Named Entity Recognition (NER) tasks, with models pre-trained on phone data showing an improvement of up to 6% F1-score above models that are trained from scratch. Pre- processing and training code will be uploaded to https://github.com/sil-ai/phone-it-in. 1 Introduction Pre-trained language models are increasingly ap- plied in ways that are agnostic to targeted down- stream tasks (Brown et al., 2020). This usage has led to a proliferation of large language models trained on enormous amounts of data. For exam- ple, the recent Megatron-Turing NLG 530B model was trained on the Pile, which includes 800GB+ of text (Gao et al., 2021), and other large language models utilize large portions of the 200TB+ com- mon crawl data.1 These large data sets include impressive amounts of text, but all languages are not represented equally (or at all) in that text. The reality is that only a negligible fraction of the 7000+ currently spoken languages (Eberhard et al., 2021) have sufficient text corpora to train state-of-the- art language models. This data scarcity results in systematic inequalities in the performance of NLP tasks across the world’s languages (Blasi et al., 2021). 1https://commoncrawl.org/ Local language communities that are working to develop and preserve their languages are producing diverse sets of data beyond pure text. The Bloom Library project,2 for example, is being used by lo- cal language communities to create and translate "shell" or "template" books into many languages (426 languages at the time this paper is being writ- ten). However, Bloom allows users to do more than just translate text. Users are also recording audio tracks and sign language videos, which has resulted in 1600+ oral translations. Other examples showing the multi-modal nature of data in local lan- guages include: (i) the creation of ChoCo: a mul- timodal corpus of the Choctaw language (Brixey and Artstein, 2021); (ii) SIL International’s 50+ year effort to document endangered Austronesian languages via text, audio, and video (Quakenbush, 2007); (iii) the grassroots Masakhane effort cat- alyzing the creation and use of diverse sets of African language data (∀ et al., 2020); and (iv) work with the Me’phaa language of western Mexico that is producing digital recordings (video and audio) along with …

Content truncated to 500 words