Fermi at SemEval-2019 Task 8: An elementary but effective approach to Question Discernment in Community QA Forums

Proceedings of the 13th International Workshop on Semantic Evaluation (SemEval-2019), pages 1160–1164 Minneapolis, Minnesota, USA, June 6–7, 2019. ©2019 Association for Computational Linguistics 1160 Fermi at SemEval-2019 Task 8: An elementary but effective approach to Question Discernment in Community QA Forums Bakhtiyar Syed1, Vijayasaradhi Indurthi1,3, Manish Shrivastava1, Manish Gupta1,2, Vasudeva Varma1 1 IIIT Hyderabad, 2 Microsoft, 3 Teradata 1{syed.b, vijaya.saradhi}@research.iiit.ac.in 1{m.shrivastava, manish.gupta, vv}@iiit.ac.in 2gmanish@microsoft.com 3vijayasaradhi.indurthi@teradata.com Abstract Online Community Question Answering Fo- rums (cQA) have gained massive popularity within recent years. The rise in users for such forums have led to the increase in the need for automated evaluation for question com- prehension and fact evaluation of the answers provided by various participants in the forum. Our team, Fermi, participated in sub-task A of Task 8 at SemEval 2019 - which tackles the first problem in the pipeline of factual evalu- ation in cQA forums, i.e., deciding whether a posed question asks for a factual informa- tion, an opinion/advice or is just socializing. This information is highly useful in segregat- ing factual questions from non-factual ones which highly helps in organizing the questions into useful categories and trims down the prob- lem space for the next task in the pipeline for fact evaluation among the available an- swers. Our system uses the embeddings ob- tained from Universal Sentence Encoder com- bined with XGBoost for the classification sub- task A. We also evaluate other combinations of embeddings and off-the-shelf machine learn- ing algorithms to demonstrate the efficacy of the various representations and their combina- tions. Our results across the evaluation test set gave an accuracy of 84% and received the first position in the final standings judged by the organizers. 1 Introduction The massive rise in popularity of Community Question Answering (cQA) forums like Stack- Overflow, Quora, Yahoo! Answers and Google Groups have led to an effective means of infor- mation dissemination for topic-centered commu- nities to share and engage in knowledge consump- tion needs. After a considerable time, information becoming obsolete is a major problem which re- sults in change of many of the facts that were pre- viously true. Another problem is that most of the forums lack exhaustive moderation and control – which results in high-latency quality checks and eventually results in the sharing of non-factual in- formation. Various factors are responsible for this – primarily being ignorance or misunderstanding and sometimes, maliciousness of the responder to the questions (Mihaylova et al., 2018). In the pipeline of detection of whether the given responses to a question are indeed factual, the nec- essary first step is to discern what category the question asked in the cQA forum falls into. As an example, “What is Domino’s customer service number?” is a factual question as it asks for a fact rather than an opinion or discourse. In contrast, consider the question “Can someone recommend a good pediatrician in Mumbai?” asks for an opin- ion rather than a particular factual information as opinions on the matter of a good pediatrician may be subjective …

Content truncated to 500 words