Revisiting Syllables in Language Modelling and Their Application on Low-Resource Machine Translation

Proceedings of the 29th International Conference on Computational Linguistics, pages 4258–4267 October 12–17, 2022. 4258 Revisiting Syllables in Language Modelling and their Application on Low-Resource Machine Translation Arturo Oncevayϵ,χ Kervy Dante Rivas Rojasρ,α Liz Karen Chavez Sanchezχ Roberto Zariquieyρ,χ ϵSchool of Informatics, University of Edinburgh, Scotland ρPontificia Universidad Católica del Perú (αIA-PUCP | χChana Field Station), Peru a.oncevay@ed.ac.uk,rzariquiey@pucp.edu.pe Abstract Language modelling and machine translation tasks mostly use subword or character inputs, but syllables are seldom used. Syllables pro- vide shorter sequences than characters, re- quire less-specialised extracting rules than mor- phemes, and their segmentation is not impacted by the corpus size. In this study, we first explore the potential of syllables for open- vocabulary language modelling in 21 languages. We use rule-based syllabification methods for six languages and address the rest with hyphen- ation, which works as a syllabification proxy. With a comparable perplexity, we show that syllables outperform characters and other sub- words. Moreover, we study the importance of syllables on neural machine translation for a non-related and low-resource language-pair (Spanish–Shipibo-Konibo). In pairwise and multilingual systems, syllables outperform un- supervised subwords, and further morpholog- ical segmentation methods, when translating into a highly synthetic language with a trans- parent orthography (Shipibo-Konibo). Finally, we perform some human evaluation, and dis- cuss limitations and opportunities. 1 Introduction In language modelling (LM), we learn distributions over sequences of words, subwords or characters, and the last two can allow an open-vocabulary gen- eration (Sutskever et al., 2011). We rely on sub- word segmentation as a widespread approach to generate rare subword units (Sennrich et al., 2016). However, the lack of a representative corpus, in terms of the word vocabulary, can constrain the unsupervised segmentation (e.g. with scarce mono- lingual texts (Joshi et al., 2020)). As an alternative, we could use character-level modelling, since it also has access to subword information (Kim et al., 2016), but we face long-term dependency issues and require longer training time to converge. Sim- ilar issues are extended to other generation tasks, such as machine translation (MT). In this context, we focus on syllables, which are speech units: “A syl-la-ble con-tains a sin-gle vow- el u-nit”. syllables can be defined as a group of segments that is pronounced as a single articulatory movement. Syllables are fundamental phonologi- cal units since they participate in important word prosodic patterns, such as stress assignment. In this sense, syllables are more linguistically relevant units than characters, and behave as a mapping function to reduce the length of the sequence with a larger “alphabet” or syllabary. Their extraction can be rule-based and corpus-independent, but data- driven methods or hyphenation using dictionaries can approximate them as well. We assess whether syllables are useful for en- coding and/or decoding a diverse set of languages on two generation tasks. First, for LM, we study 21 languages, to cover different levels of ortho- graphic depth, which is the degree of grapheme- phoneme correspondence (Borgwaldt et al., 2005) and a factor that can increase complexity to syl- labification (Marjou, 2021).1 …

Content truncated to 500 words