Combining Noisy Semantic Signals with Orthographic Cues: Cognate Induction for the Indic Dialect Continuum

Proceedings of the 26th Conference on Computational Natural Language Learning (CoNLL), pages 110 - 131 December 7-8, 2022 ©2022 Association for Computational Linguistics Combining Noisy Semantic Signals with Orthographic Cues: Cognate Induction for the Indic Dialect Continuum Niyati Bafna1,3,4∗, Josef van Genabith1,2, Cristina España-Bonet2, Zdenˇek Žabokrtský3 1Saarland Informatics Campus, Saarland University, Germany 2DFKI GmbH, Germany 3Institute of Formal and Applied Linguistics, Charles University, Prague 4Inria, Paris, France niyatibafna13@gmail.com,{josef.van_genabith,cristinae}@dfki.de, zabokrtsky@ufal.mff.cuni.cz Abstract We present a novel method for unsupervised cognate/borrowing identification from mono- lingual corpora designed for low and extremely low resource scenarios, based on combining noisy semantic signals from joint bilingual spaces with orthographic cues modelling sound change. We apply our method to the North Indian dialect continuum, containing several dozens of dialects and languages spoken by more than 100 million people. Many of these languages are zero-resource and therefore nat- ural language processing for them is non- existent. We first collect monolingual data for 26 Indic languages, 16 of which were previ- ously zero-resource, and perform exploratory character, lexical and subword cross-lingual alignment experiments for the first time at this scale on this dialect continuum. We create bilin- gual evaluation lexicons against Hindi for 20 of the languages. We then apply our cognate iden- tification method on the data, and show that our method outperforms both traditional orthogra- phy baselines as well as EM-style learnt edit distance matrices. To the best of our knowl- edge, this is the first work to combine tradi- tional orthographic cues with noisy bilingual embeddings to tackle unsupervised cognate de- tection in a (truly) low-resource setup, showing that even noisy bilingual embeddings can act as good guides for this task. We release our mul- tilingual dialect corpus, called HinDialect, as well as our scripts for evaluation data collection and cognate induction.2 1 Introduction Hindi is listed as one of the 22 official languages of India, with the latest census showing 43.63% of Indians as having Hindi as their mother tongue.3 *This work was done at Charles University and Saarland University as a Masters’ Thesis. 2See http://hdl.handle.net/11234/1-4839 and https://github.com/niyatibafna/ north-indian-dialect-modelling, respectively. 3https://en.wikipedia.org/wiki/2011_Census_of_ India However, this figure counts speakers of the lan- guages of the whole Indic/Indo-Aryan (IA) dialect continuum, the “Hindi Belt”, that stretches from Rajasthan in the West to Bihar and Jharkhand in the East, and of which modern standard Hindi is only a part.4 This continuum, spread out over North and Central India, contains a wide variety of lan- guages/dialects that may even be mutually incom- prehensible, and form subfamilies of their own, e.g. the Rajasthani, Bihari, or Pahari subfamilies.5 Natural language processing (NLP) resources for these languages are sorely lacking; most of these languages, despite having millions of speak- ers, have little or no monolingual data, no linguistic resources such as lexicons, grammars, taggers, let alone more elaborate resources such as parallel data or pretrained embeddings. We focus on 26 languages of the Hindi Belt writ- ten in the Devanagari script and make the following contributions: (i) we collect the first monolingual resources …

Content truncated to 500 words