Multilingual unsupervised sequence segmentation transfers to extremely low-resource languages

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 5331 - 5346 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Multilingual unsupervised sequence segmentation transfers to extremely low-resource languages C.M. Downey, Shannon Drizin∗, Levon Haroutunian∗, Shivin Thukral∗ Department of Linguistics, University of Washington {cmdowney, sdrizin, levonh, shivin7}@uw.edu Abstract We show that unsupervised sequence- segmentation performance can be transferred to extremely low-resource languages by pre- training a Masked Segmental Language Model (Downey et al., 2021) multilingually. Further, we show that this transfer can be achieved by training over a collection of low-resource languages that are typologically similar (but phylogenetically unrelated) to the target language. In our experiments, we transfer from a collection of 10 Indigenous American languages (AmericasNLP, Mager et al., 2021) to K’iche’, a Mayan language. We compare our multilingual model to a monolingual (from-scratch) baseline, as well as a model pre-trained on Quechua only. We show that the multilingual pre-trained approach yields consistent segmentation quality across target dataset sizes, exceeding the monolingual baseline in 6/10 experimental settings. Our model yields especially strong results at small target sizes, including a zero-shot performance of 20.6 F1. These results have promising implications for low-resource NLP pipelines involving human-like linguistic units, such as the sparse transcription framework proposed by Bird (2020). 1 Introduction Unsupervised sequence segmentation (at the word, morpheme, and phone level) has long been an area of interest in languages without whitespace- delimited orthography (e.g. Chinese, Uchiumi et al., 2015; Sun and Deng, 2018), morphologi- cally complex languages without rule-based mor- phological analyzers (Creutz and Lagus, 2002), and automatically phone-transcribed speech data (Goldwater et al., 2009; Lane et al., 2021), respec- ∗Equal contribution from starred authors, sorted by last name. Sincere thanks to: Gina-Anne Levow, Shane Steinert Threlkeld, and Sara Ng for helpful comments and discussion; Francis Tyers for access to the K’iche’ data; Manuel Mager for access to the morphologically-segmented validation data tively. It has been particularly important for lower- resource languages in which there is little or no gold-standard data on which to train supervised models (Joshi et al., 2020). In modern neural end-to-end systems, unsu- pervised segmentation is usually performed via information-theoretic algorithms such as BPE (Sen- nrich et al., 2016) and SentencePiece (Kudo and Richardson, 2018). However, the segmentations they produce are largely non-sensical to humans (Park et al., 2021). The motivating tasks listed above instead require unsupervised approaches that correlate more closely with human judgements of the boundaries of linguistic units. For exam- ple, in a human-in-the-loop framework such as the sparse transcription proposed by Bird (2020), lexical items are automatically proposed to native speakers for confirmation, and it is important that these candidates be (close to) sensical, recogniz- able pieces of language. In this paper, we investigate the utility of re- cent models that have been developed to con- duct unsupervised surface morpheme segmenta- tion as a byproduct of a language modeling objec- tive (e.g. Kawakami et al., 2019; Downey et al., 2021, see Section 2). The …

Content truncated to 500 words