Using a Word-Space Model to Determine the Relevance of Messages in Anchored Asynchronous Online Discussions

Using a Word-Space Model to Determine the Relevance of Messages in Anchored Asynchronous Online Discussions * Rodolfo Ragaa, Jennifer Ragaa, Erick Bonusa, and Raymund Sisonb aComputer Science Department, Jose Rizal University, 80 Shaw Boulevard, Mandaluyong City, Philippines bCollege of Computer Studies, De La Salle University-Manila, 2401 Taft Avenue, Manila, Philippines {jundy.raga, jennie.raga,erick.bonus}@gmail.com; sisonr@dlsu.edu.ph Abstract. This paper presents results of the first phase of our study aimed at investigating the applicability of word-space models, particularly those generated using Random Indexing (RI) technique, for the task of determining the relevance of messages posted in anchored asynchronous online discussion forums. In this phase, we addressed several questions intended to establish baseline figures: How efficient will the word-space model perform in this task? How much of its classification decisions align with the decisions of the human annotators? More importantly, does the paradigmatic and syntagmatic contexts of words have a direct effect on its output and which produces the best results? Using Cohen’s Kappa and Holsti’s Coefficient Reliability measure, our experiments generated an initial reliability performance (K=0.41, CR=0.72). It further indicated that the syntagmatic context is more applicable for this task. We concluded with a discussion of the weaknesses identified and possible means of improving the level of performance. Keywords: Asynchronous Online Discussion, Word-Space Model, Random Indexing, Paradigmatic Context, Syntagmatic Context. 1. Introduction Asynchronous Online Discussions (AOD), also known as forums or threaded discussion boards, are a popular form of web-based computer-mediated communication (CMC). More recently, this form of communication has come into focus in education as an alternative form of assessment that can be used to supplement traditional classroom discussions. The advantage of this communication medium lies in its anytime and anywhere accessibility which provides logistic flexibility for students to communicate with each other and for teachers to assess their interaction. A disadvantage of this medium however is that the focus of discussion is prone to topic drifting; this drifting is primarily due to the time-lags inherent in the asynchronous mode of discussion. Another problem is that the manual monitoring and assessment of message contributions are time-consuming and often tedious, especially for teachers. For these reasons, automated analysis for discussion understanding and better methods of topic-focus monitoring to enable more efficient information assessment are much sought for (Ravi & Kim, 2007). In our study, we aim to investigate the feasibility of utilizing the functionality of word-space models for the task of analyzing online discussion transcripts. More particularly, we want to determine how word-space models can be used to detect the relevance of individual message posts Copyright 2008 by Rodolfo Raga, Jennifer Raga, Eric Bonus, and Raymund Sison 321 22nd Pacific Asia Conference on Language, Information and Computation, pages 321–330 relative to the overall topic of discussion. Several methods for generating word-space models are available and have been successfully tested on detecting similarity between terms in groups of texts, but to our knowledge very few have been applied and tested on analyzing online discussion transcripts, a rare example is that of (McArthur and Bruza, 2003) …

Content truncated to 500 words