Evaluating the Impact of Sub-word Information and Cross-lingual Word Embeddings on Mi’kmaq Language Modelling

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 2736–2745 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 2736 Evaluating the Impact of Sub-word Information and Cross-lingual Word Embeddings on Mi’kmaq Language Modelling Jeremie Boudreau,1 Akankshya Patra,2 Ashima Suvarna1 and Paul Cook1 1. Faculty of Computer Science, University of New Brunswick 2. University of Southern California jeremie@boudreau.me, patra@usc.edu, asuvarna31@gmail.com, paul.cook@unb.ca Abstract Mi’kmaq is an Indigenous language spoken primarily in Eastern Canada. It is polysynthetic and low-resource. In this paper we consider a range of n-gram and RNN language models for Mi’kmaq. We find that an RNN language model, initialized with pre-trained fastText embeddings, performs best, highlighting the importance of sub-word information for Mi’kmaq language modelling. We further consider approaches to language modelling that incorporate cross-lingual word embeddings, but do not see improvements with these models. Finally we consider language models that operate over segmentations produced by SentencePiece — which include sub-word units as tokens — as opposed to word-level models. We see improvements for this approach over word-level language models, again indicating that sub-word modelling is important for Mi’kmaq language modelling. Keywords: Indigenous languages, language modelling, word embeddings 1. Introduction Mi’kmaq is an Indigenous language spoken primarily in Eastern Canada (Johnson, 1996). It is polysynthetic and verb-oriented, and in the Eastern Algonquian language family. Mi’kmaq has roughly 8,000 speakers in Canada,1 and is a low-resource language. There are Mi’kmaq dic- tionaries (Rand, 1888; DeBlois, 1996) and translated texts (DeBlois, 1990), but no large corpora. There has been very little prior computational work on Mi’kmaq, with the ex- ception of Maheshwari et al. (2018), who built a web cor- pus of Mi’kmaq and carried out preliminary language mod- elling experiments using this corpus. Language models are a crucial component for many language technology systems including spelling correc- tion and word suggestion on smartphone soft keyboards. Mi’kmaq language modelling is, however, particularly challenging due to its rich morphology and the relatively small amount of data available. Mi’kmaq is polysynthetic, so each word is composed of many morphemes (Johnson, 1996). Rand (1888, p. iv) explains that a single Mi’kmaq word can essentially describe a whole English sentence. Language models are often trained on billions of tokens of text (Chelba et al., 2014; Merity et al., 2017; Jozefowicz et al., 2016), but the Mi’kmaq corpus built by Maheshwari et al. (2018) is only 76k tokens. The rich morphology of Mi’kmaq suggests that language models that operate only at the word level, and do not model the internal structure of words, might perform poorly. Moreover, because of this rich morphology, we ex- pect many out-of-vocabulary (OOV) words, and therefore it is important that a Mi’kmaq language model be able to handle OOVs. In this paper we consider n-gram and RNN language mod- els for Mi’kmaq. We tune these models over a range of pa- rameters in an effort to establish a strong baseline. We then 1https://www12.statcan.gc.ca/ census-recensement/2011/as-sa/98-314-x/ 98-314-x2011003_3-eng.cfm consider the use of pre-trained word embeddings to …

Content truncated to 500 words