How low is too low? A monolingual take on lemmatisation in Indian languages

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4088–4094 June 6–11, 2021. ©2021 Association for Computational Linguistics 4088 How low is too low? A monolingual take on lemmatisation in Indian languages Kumar Saunack∗, Kumar Saurav∗, Pushpak Bhattacharyya IIT Bombay {krsaunack, krsrv, pb}@cse.iitb.ac.in Abstract Lemmatization aims to reduce the sparse data problem by relating the inflected forms of a word to its dictionary form. Most prior work on ML based lemmati- zation has focused on high resource lan- guages, where data sets (word forms) are readily available. For languages which have no linguistic work available, espe- cially on morphology or in languages where the computational realization of linguistic rules is complex and cumbersome, machine learning based lemmatizers are the way to go. In this paper, we devote our attention to lemmatisation for low resource, morpho- logically rich scheduled Indian languages using neural methods. Here, low resource means only a small number of word forms are available. We perform tests to analyse the variance in monolingual models’ perfor- mance on varying the corpus size and con- textual morphological tag data for train- ing. We show that monolingual approaches with data augmentation can give competi- tive accuracy even in the low resource set- ting, which augurs well for NLP in low re- source setting. 1 Introduction Natural Language Processing (NLP) has seen remarkable growth in all its sub-areas like ma- chine translation, summarization, question an- swering and so on. For all these tasks, though, morphemes remain the most basic form of in- formation (Otter et al., 2020). Morpheme identification (lemma and affixes) can assist these very useful large applications by solving the data sparsity problem. Good lemmatisers are invaluable tools for handling large vocabulary in morphologically rich languages and thereby boosting perfor- mance in downstream tasks, but techniques *These authors contributed equally to this work are limited by resource availability. This is a relevant point for Indian languages. For in- stance, as many as 197 Indian languages are in the UNESCO’s Atlas of the ”World’s Lan- guages in Danger, 2010”. Even among the 22 scheduled languages of India, there is a wide disparity in resource availability, for example, for Konkani and Kashmiri (Rajan et al., 2020; Islam et al., 2018). Techniques like Porter stemmer are indeed quick solutions, but they are suited only for alphabetic script languages, like English, and not abugida, like Bengali (Ali et al., 2017), or abjad, like Urdu (Kansal et al., 2012), script languages. Moreover, creating stemmers re- quires different language specific stemming al- gorithms. This requirement of language spe- cific measures comes in the way of scaling the enterprise of creating stemmers for the hun- dreds and thousands of languages that exist in the world. One might think of ML for stemming- for example, training a neural net with stems and word forms; but almost none of the 22 scheduled Indian languages, which is just a subset of the numerous languages spo- ken and written in …

Content truncated to 500 words