Assessing Users’ Reputation from Syntactic and Semantic Information in Community Question Answering

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 5383–5391 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 5383 Assessing Users’ Reputation from Syntactic and Semantic Information in Community Question Answering Yonas Woldemariam Dept. Computing Science, Umeå University, Sweden yonasd@cs.umu.se Abstract Textual content is the most significant as well as substantially the big part of CQA forums. Users gain reputation for contributing such content. Although linguistic quality is the very essence of textual information, that does not seem to be considered in estimating users’ reputation. As existing users’ reputation systems seem to solely rely on vote counting, adding that bit of linguistic information surely improves their quality. In this study, we investigate the relationship between users’ reputation and linguistic features extracted from their associated answers content. And we build statistical models on a Stack Overflow dataset that learn reputation from complex syntactic and semantic structures of such content. The resulting models reveal how users’ writing styles in answering questions play important roles in building reputation points. In our experiments, extracting answers from systematically selected users followed by linguistic features annotation and models building. The models are evaluated on in-domain (e.g., Server Fault, Super User) and out-domain (e.g., English, Maths) datasets. We found out that the selected linguistic features have quite significant influences over reputation scores. In the best case scenario, the selected linguistic feature set could explain 80% variation in reputation scores with the prediction error of 3%. The per- formance results obtained from the baseline models have been significantly improved by adding syntactic and punctuation marks features. Keywords: Reputation Prediction, Community Question-Answering, Competence Analysis, Syntax and Semantics 1. Introduction Although many CQA forums aim to provide reasonably high-quality content, their underlying mechanisms to con- trol the quality and reliability of such content lack lots of important features (Shah and Pomerantz, 2010). Particu- larly, linguistic features are quite pertinent to the very na- ture of such content, as textual content is the most signif- icant as well as substantially the big part of CQA forums. Moreover, even if it is quite apparent how linguistic qual- ity contributes for gaining reputation, existing reputation systems do not seem to take such quality into account in reputation scores estimation. General CQAs’ data usage trends show that the number of users actively visiting such sites and the rate at which users regularly posting content are dramatically increasing (Bal- tadzhieva and Chrupala, 2015b). For instance, right now, the StackExchange (SE) Stack Overflow’s traffic shows it is receiving nearly 10 million page views and 6.4k ques- tions per day, there are also many forums in SE with al- most 100% question answering rate. Despite such huge amount of data are consumed by constantly growing num- ber of users, the data suffers from serious quality (e.g., in- accuracies and grammatical errors) and provenance (e.g., data ownership and trust- worthiness) issues that potentially mislead peoples looking for right answers for their ques- tions. In many CQA sites including SE, …

Content truncated to 500 words