A Hybrid Feature Set based Maximum Entropy Hindi Named Entity Recognition

A Hybrid Feature Set based Maximum Entropy Hindi Named Entity Recognition Sujan Kumar Saha Indian Institute of Technology Kharagpur, West Bengal India - 721302 sujan.kr.saha@gmail.com Sudeshna Sarkar Indian Institute of Technology Kharagpur, West Bengal India - 721302 shudeshna@gmail.com Pabitra Mitra Indian Institute of Technology Kharagpur, West Bengal India - 721302 pabitra@gmail.com Abstract We describe our effort in developing a Named Entity Recognition (NER) system for Hindi using Maximum Entropy (Max- Ent) approach. We developed a NER an- notated corpora for the purpose. We have tried to identify the most relevant features for Hindi NER task to enable us to develop an efficient NER from the limited corpora developed. Apart from the orthographic and collocation features, we have experimented on the efficiency of using gazetteer lists as features. We also worked on semi-automatic induction of context patterns and experi- mented with using these as features of the MaxEnt method. We have evaluated the per- formance of the system against a blind test set having 4 classes - Person, Organization, Location and Date. Our system achieved a f-value of 81.52%. 1 Introduction Named Entity Recognition involves locating and classifying the names in text. NER is an important task, having applications in Information Extraction (IE), question answering, machine translation and in most other NLP applications. NER systems have been developed for English and few other languages with high accuracies. These systems take advantage of large amount of Named Entity (NE) annotated corpora and other NER re- sources. However when we started working on a NER system for Hindi, we did not have any NER annotated corpora for Hindi, neither did we have ac- cess to any comprehensive gazetteer list. In this work we have identified suitable features for the Hindi NER task. Orthography features, the suffix and prefix information, as well as information about the sorrounding words and their tags are used to develop a Maximum Entropy (MaxEnt) based Hindi NER system. Additionally, we have acquired gazetteer lists for Hindi and used these gazetteers in the Maximum Entropy (MaxEnt) based Hindi NER system. We also worked on semi-automatically learning of context pattern for identifying names. These context pattern rules have been integrated into the MaxEnt based NER system, leading to a high ac- curacy. The paper is organized as follows. A brief survey of different techniques used for the NER task in dif- ferent languages and domains are presented in Sec- tion 2. The MaxEnt based NER system is described in Section 3. Various features used in NER are then discussed. Next we present the experimental results and related discussions. Finally Section 8 concludes the paper. 2 Previous Work A variety of techniques has been used for NER. The two major approaches to NER are: 1. Linguistic approaches. 2. Machine Learning based approaches. The linguistic approaches typically use rules man- ually written by linguists. There are several rule- based NER systems, containing mainly lexicalized 343 grammar, gazetteer lists, and list of trigger words, which are capable of providing 88%-92% f-measure accuracy for English …

Content truncated to 500 words