Study of Similarity Measures as Features in Classification for Answer Sentence Selection Task in Hindi Question Answering: Language-Specific v/s Other Measures

Study of Similarity Measures as Features in Classification for Answer Sentence Selection Task in Hindi Question Answering: Language-Specific v/s Other Measures Devika A Verma BITS Pilani K K Birla Goa Campus p20160010@ goa.bits-pilani.ac.in Ramprasad S Joshi BITS Pilani K K Birla Goa Campus rsj@ goa.bits-pilani.ac.in Shubhamkar A Joshi Vishwakarma Institute of Information Technology shubhamkar.21810437@ viit.ac.in Onkar K Susladkar Vishwakarma Institute of Information Technology onkar.21810471@ viit.ac.in Abstract Answer sentence selection is an important sub-task in Question Answering (QA) that de- termines the correct answer sentence from a passage. This task can naturally be reduced to the semantic text similarity problem between question and answer candidate. In this work, we investigate the significance of various sim- ilarity measures for the answer sentence se- lection task in Hindi an Indo-Aryan language. Karaka relations is the core of dependency an- notation scheme used for Hindi and are cru- cial to syntactico-semantic analysis of the sen- tence. We investigate this, and compare them to other, hitherto known measures. To inves- tigate and compare the utility of various mea- sures, we develop a test-bench over a bench- mark Hindi and English multilingual QA cor- pus for comparison, making two tool-chains and designing empirical experiments across combinations of similarity measures, sentence embedding schemes, and supervised machine learning models for classification. Combin- ing Karaka relations with different similarity measures shows significant performance im- provement for sentence selection task, sug- gesting them as potentially a semantic simi- larity measure. Moreover, our results give us confidence that refinement of Karaka relations extraction to optimal quality will reduce the need for availability of large pre-trained lan- guage models. 1 Introduction Information-retrieval-based (IR-based) QA systems typically employ a search engine and a passage re- trieval module that narrows down the answer search space from a huge corpus to a small set of sentences. On the contrary, QA systems depending on reading comprehension are provided with a passage for an- swer extraction. In both kinds of systems, the re- turned answer can be a word, a phrase or a sentence. Wang et al. (2007) highlighted the importance of re- turning a complete sentence over a short phrase as an answer against the user query. The study conducted by Lin et al. (2003) shows that as the amount of text returned by a QA system increases, users utilize the supporting text to get the answer, significantly de- creasing the number of further queries. Answer sentence selection as an important task in QA has been a widely researched topic since the release of QASent and WikiQA corpora. QASent (Wang et al., 2007) was created by choosing sen- tences from Text REtrieval Conference (TREC) QA 8-13 track data(Voorhees, 1999). WikiQA (Yang et al., 2015), that was released later, was developed in a more natural and large setting. Researchers have sought to accomplish this task by combining (in varying degrees) techniques of natural language processing, statistical analysis and deep learning for lexical, syntactic and semantic processing of the question and candidate sentences. We can certainly say that …

Content truncated to 500 words