Measuring Semantic Similarity for Bengali Tweets Using WordNet

Proceedings of Recent Advances in Natural Language Processing, pages 537–544, Hissar, Bulgaria, Sep 7–9 2015. Measuring Semantic Similarity for Bengali Tweets Using WordNet Dwijen Rudrapal CSE Department NIT Agartala, India dwijen.rudrapal@gmail.com Amitava Das CSE Department IIIT, Sricity, India amitava.santu@gmail.com Baby Bhattacharya CSE Department NIT Agartala, India babybhatt75@gmail.com Abstract Similarity between natural language texts, sentences in terms of meaning, known as tex- tual entailment, is a generic problem in the area of computational linguistics. In the last few years researchers worked on various as- pects of textual entailment problem, but mostly restricted to English language. Here in this paper we present a method for measuring the semantic similarity of Bengali tweets us- ing WordNet. Moreover we defined partial textual entailment (PTE) as in real data par- tial entailment cases are equally prevalent with the complete/direct entailment. Alt- hough by definition entailment is a direction- al relationship, but here we consider entail- ment more as semantic similarity. Keywords: Semantic similarity; WordNet; Synonym; 1 Introduction Variations of natural language expression make it difficult to determine semantically equivalent sentences. The beauty of natural languages is similar meaning could be expressed in countless ways; therefore it is a very complex task to measure relatedness of natural language sentenc- es. Morpho-Syntactic variations of similar mean- ing expressions are more prevalent in social me- dia text due to its informal nature. Semantic similarity score plays important role in many Natural Language Applications (NLP) such as multi-document summarization (MDS), question answering(QA), information extraction(IE) (Bhagwani et al., 2012). Several researchers have explored numbers of semantic similarity methods mostly for English but very less for Indian lan- guages and almost nothing for Bengali. Techni- cally these methods can be categorized into two groups: dictionary/thesaurus-based (one such example is edge counting-based) methods and corpus-based (one such method is information theory-based) methods (Li et al., 2003). Edge counting based methods use only semantic links and corpus based methods combine corpus statis- tics with taxonomic distances. The objective of this work is to design a sys- tem to measure semantic similarity score be- tween two Bengali tweets. We adopted a lexical based method; the words are grouped into clus- ters in terms of their senses along with their syn- onyms. Our proposed method centered on ana- lyzing shared words similarity among tweets. Partial Textual Entailment (PTE) is defined as a bidirectional relationship among a sen- tence/tweet pair. It defines partial/complete meaning inference from one sentence/text from another text. We define these following 4 de- tailed PTE categories: 1. Type 1: If both the given texts are having same information and mean same, then it is a case of direct entailment and should be noted as (X=X). 2. Type 2: If the first/second given text has any extra information than the second/first text respectively then it is been categorized as PTE2. This type may have two variations like: (X=X+Z or X+Z=X). 3. Type 3: If the first given text has all the in- formation of the second given text and has some extra information, …

Content truncated to 500 words