A Neural Lemmatizer for Bengali

A Neural Lemmatizer for Bengali Abhisek Chakrabarty, Akshay Chaturvedi, Utpal Garain CVPR Unit, Indian Statistical Institute 203 B.T. Road, Kolkata-700108, India abhisek0842@gmail.com, thegphenomenon@gmail.com, utpal@isical.ac.in Abstract We propose a novel neural lemmatization model which is language independent and supervised in nature. To handle the words in a neural framework, word embedding technique is used to represent words as vectors. The proposed lemmatizer makes use of contextual information of the surface word to be lemmatized. Given a word along with its contextual neighbours as input, the model is designed to produce the lemma of the concerned word as output. We introduce a new network architecture that permits only dimension specific connections between the input and the output layer of the model. For the present work, Bengali is taken as the reference language. Two datasets are prepared for training and testing purpose consisting of 19, 159 and 2, 126 instances respectively. As Bengali is a resource scarce language, these datasets would be beneficial for the respective research community. Evaluation method shows that the neural lemmatizer achieves 69.57% accuracy on the test dataset and outperforms the simple cosine similarity based baseline strategy by a margin of 1.37%. Keywords: Bengali, lemmatization, neural network 1. Introduction In this article, we propose a novel neural network based lemmatization method and show its effectiveness for a highly inflected language namely, Bengali. Bengali belongs to the group of major Indic languages and currently finds its place among the top 10 most popular languages in the world. The very first hurdle in computational processing of major Indic languages is the presence of various morpho- logical variants of root words in the text. Existence of mor- phological richness in those languages creates problems for developing several text processing tasks like Natural Lan- guage Processing (NLP), Information Extraction (IE), In- formation Retrieval (IR) etc. For IR, stemming serves the purpose well (Majumder et al., 2007; Paik and Parui, 2011; Paik et al., 2011a; Paik et al., 2011b; Paik et al., 2013) but where semantic processing is needed (e.g. Word Sense Dis- ambiguation (WSD), Machine Translation (MT) etc.), there lemmatization stands as a mandatory pre-processing mod- ule. The role of a lemmatizer is to map a surface word in a context to its root form. In the lexical resources, the roots of the language are listed with their syntactic and semantic descriptions. Hence, determining the appropriate lemma is crucial for knowing the meaning or other linguistic proper- ties of a word in a raw text. 1.1. State of the Art So far, there has been a little research conducted on developing efficient lemmatization algorithms for Ben- gali (Faridee et al., 2009; Loponen et al., 2013; Bhat- tacharyya et al., 2014). The Bengali morphological ana- lyzer built by Faridee et al. (2009) is a rule based one and capable of lemmatizing only colloquial words in the language. Also, the lemmas for verbs produced by their method are basically stems, and consequently they can- not be linked to standard dictionaries. The work by Lo- ponen …

Content truncated to 500 words