PMI-cool at SemEval-2016 Task 3: Experiments with PMI and Goodness Polarity Lexicons for Community Question Answering

Proceedings of SemEval-2016, pages 844–850, San Diego, California, June 16-17, 2016. c⃝2016 Association for Computational Linguistics PMI-cool at SemEval-2016 Task 3: Experiments with PMI and Goodness Polarity Lexicons for Community Question Answering Daniel Balchev, Yasen Kiprov, Ivan Koychev Faculty of Mathematics and Informatics Sofia University “St. Kliment Ohridski” Sofia, Bulgaria {dblachev,yasen.kiprov}@gmail.com koychev@fmi.uni-sofia.bg Preslav Nakov Qatar Computing Research Institute HBKU, P.O. box 5825 Doha, Qatar pnakov@qf.org.qa Abstract We describe our submission to SemEval-2016 Task 3 on Community Question Answering. We participated in subtask A, which asks to rerank the comments from the thread for a given forum question from good to bad. Our approach focuses on the generation and use of goodness polarity lexicons, similarly to the sentiment polarity lexicons, which are very popular in sentiment analysis. In particular, we use a combination of bootstrapping and pointwise mutual information to estimate the strength of association between a word (from a large unannotated set of question-answer threads) and the class of good/bad comments. We then use various features based on these lexicons to train a regression model, whose predictions we use to induce the final com- ment ranking. While our system was not very strong as it lacked important features, our lex- icons contributed to the strong performance of another top-performing system. 1 Introduction Online forums have been gaining a lot of popularity in recent years. In these forums, one can ask a ques- tion, and based on the wisdom of the crowd, expect to get some good answers. In practice, unless there is strong moderation, most such forums get popu- lated with bad answers, which can be annoying for users as it takes time to read through all answers in a long thread. As the importance of the problem was recognized in the research community, this gave rise to two shared tasks on Community Question An- swering at SemEval-2015 (Nakov et al., 2015) and SemEval-2016 (Nakov et al., 2016). Here we describe the PMI-cool system, which we developed to participate in SemEval-2016 Task 3, subtask A, which asks to rerank the answers in a question-answer thread, ordering them from good to bad (Nakov et al., 2016). As the name of our system suggests, our approach is heavily based on pointwise mutual information (PMI), which we use to estimate the association strength between a word and a class, e.g., the class of good or the class of bad comments. Based on this association strength, we perform bootstrapping in a large unannotated set of question-answer threads to generate specialized goodness polarity lexicons. We then use various fea- tures based on these lexicons to train a regression model, whose predictions we use to induce the final comment ranking. While our PMI-cool system did not perform very well at the competition as it lacked important fea- tures and as we found a bug in our submission, our goodness polarity lexicons proved useful and con- tributed to the strong performance of another top- performing system at SemEval-2016 Task 3: SU- per team (Mihaylova et al., 2016). …

Content truncated to 500 words