Massively Multilingual Pronunciation Modeling with WikiPron

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 4223–4228 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 4223 Massively Multilingual Pronunciation Mining with WikiPron Jackson L. Lee, Lucas F. E. Ashby∗, M. Elizabeth Garza∗, Yeonju Lee-Sikka∗, Sean Miller∗, Alan Wong∗, Arya D. McCarthy†, Kyle Gorman∗ ∗The Graduate Center, City University of New York †Center for Language and Speech Processing, Johns Hopkins University Abstract We introduce WikiPron, an open-source command-line tool for extracting pronunciation data from Wiktionary, a collaborative multilingual online dictionary. We first describe the design and use of WikiPron. We then discuss the challenges faced scaling this tool to create an automatically-generated database of 1.7 million pronunciations from 165 languages. Finally, we validate the pronunciation database by using it to train and evaluating a collection of generic grapheme-to-phoneme models. The software, pronunciation data, and models are all made available under permissive open-source licenses. Keywords: speech, pronunciation, grapheme-to-phoneme, g2p 1. Introduction Nearly all speech technologies depend on explicit mappings between the orthographic forms of words and their pronunci- ations, represented as a sequence of phones. These mappings are constructed using digital pronunciation dictionaries, and for out-of-vocabulary words, grapheme-to-phoneme conver- sion models trained on such dictionaries. Like many lan- guage resources, pronunciation dictionaries are expensive to create and maintain, and free, large, high-quality dictionaries are only available for a small number of languages. 1.1. Prior work Given the importance of pronunciation modeling to speech technology and the dearth of freely available data, some researchers have exploited crowd-sourced pronunciation data (Ghoshal et al., 2009). One obvious source of data is Wiktionary, a collaborative multilingual online dictionary. Wiktionary has been mined for many natural language re- sources, including UniMorph, a multilingual database of morphological paradigms (Kirov et al., 2018). Schlippe et al. (2010) extract Wiktionary pronunciation data for English, French, German, and Spanish. They report that this data is both abundant and improves automatic speech recognizer performance. However, they do not release any software or data. Deri and Knight (2016) release a collection of 650,000 word-pronunciation pairs extracted from Wiktionary; once again, they do not release the associated software. 1.2. Contributions In this paper we introduce WikiPron, an open-source tool for mining pronunciation data from Wiktionary. We then describe a database of 1.7 million word/pronunciation pairs in 165 languages, both living and dead, natural and con- structed, that we mined using this tool. Finally, we use this database to perform experiments in grapheme-to-phoneme modeling. WikiPron and the full pronunciation database are hosted at a public open-source repository.1 Materials for the modeling experiments are hosted at a separate open- source repository.2 While we target a smaller number of 1https://github.com/kylebgorman/wikipron 2https://github.com/kylebgorman/ wikipron-modeling Figure 1: Pronunciation of the Spanish word enguillar ‘to wolf down’ as it appears on Wiktionary. The entry gives phonemic and phonetic transcriptions for two dialects. languages than the 531-language data set provided by Deri and Knight (2016)—we omit languages with fewer than 100 word-pronunciation pairs, and do not perform any sort of …

Content truncated to 500 words