Multiword Expressions Dataset for Indian Languages

Multiword Expressions Dataset for Indian Languages Dhirendra Singh, Sudha Bhingardive, Pushpak Bhattacharyya Department of Computer Science and Engineering, Indian Institute of Technology Bombay, India. {dhirendra,sudha,pb}@cse.iitb.ac.in Abstract Multiword Expressions (MWEs) are used frequently in natural languages, but understanding the diversity in MWEs is one of the open problem in the area of Natural Language Processing. In the context of Indian languages, MWEs play an important role. In this paper, we present MWEs annotation dataset created for Indian languages viz., Hindi and Marathi. We extract possible MWE candidates using two repositories: 1) the POS-tagged corpus and 2) the IndoWordNet synsets. Annotation is done for two types of MWEs: compound nouns and light verb constructions. In the process of annotation, human annotators tag valid MWEs from these candidates based on the standard guidelines provided to them. We obtained 3178 compound nouns and 2556 light verb constructions in Hindi and 1003 compound nouns and 2416 light verb constructions in Marathi using two repositories mentioned before. This created resource is made available publicly and can be used as a gold standard for Hindi and Marathi MWE systems. Keywords: Multiword Expressions, MWEs, WordNet, Hindi WordNet, Compound Nouns, Light Verb Construc- tions 1. Introduction Recently, various approaches have been proposed for the identification and extraction of MWEs (Cal- zolari et al., 2002; Baldwin et al., 2003; Guevara, 2010; Al-Haj and Wintner, 2010; Kunchukuttan and Damani, 2008; Chakrabarti et al., 2008; Sinha, 2011; Singh et al., 2012; Reddy et al., 2011). The quality of such approaches depends on the use of algorithms and also on the quality of resources used. Various standard MWEs datasets1 are available for languages like English, French, German, Portuguese, etc and can be used for evaluation of MWE approaches. But for Indian languages, no such standard datasets are available publicly. Our goal is to create MWEs annotation for Indian languages viz., Marathi and make it available publicly.Hindi and We have explored two types of MWEs: compound nouns (CNs) and light verb constructions (LVCs), since they are used very frequently in the text data in comparison to other MWEs. The created resource can be useful for various natural language processing applications like information extraction, word sense disambiguation, machine translation, etc. The rest of the paper is organized as follows. Section 2 gives detail about the compound nouns and light verb constructions. Section 3 describes the extraction process of possible MWE candidates. Section 4 gives the statistics of MWEs annotation for Hindi and Marathi. MWEs guidelines are given in Section 5 followed by discussions in Section 6. Section 7 concludes the paper and points to the future work. 1http://multiword.sourceforge.net/PHITE.php?sitesig= FILES&page =FILES_20_Data_Sets 2. Compound Nouns and Light Verb Constructions In the context of Indian languages, MWEs are quite varied and many of these are borrowed from other lan- guages like English, Urdu, Arabic, Sanskrit, etc. For Hindi, there are limited investigations on MWE extrac- tion. Venkatapathy et. al., (2006) worked on syntac- tic and semantic features for N-V collocation extrac- tion using MaxEnt classifier. Mukerjee et al., …

Content truncated to 500 words