An Initial Test Collection for Ranked Retrieval of SMS Conversations

An Initial Test Collection for Ranked Retrieval of SMS Conversations Rashmi Sankepally, Douglas W. Oard iSchool and UMIACS University of Maryland College Park, MD USA 20742 rashmi@umd.edu, oard@umd.edu Abstract This paper describes a test collection for evaluating systems that search English SMS (Short Message Service) conversations. The collection is built from about 120,000 text messages. Topic development involved identifying typical types of information needs, then generating topics of each type for which relevant content might be found in the collection. Relevance judgments were then made for groups of messages that were most highly ranked by one or more of several ranked retrieval systems. The resulting TREC style test collection can be used to compare some alternative retrieval system designs. Keywords: ranked retrieval, test collection, relevance assessment, retrieval effectiveness 1. Introduction With the ubiquity of smartphone usage, many people have turned to using short message services (SMS) to send quick messages that don’t require an immediate response. As with other types of “conversational text” (e.g., email, dis- cussion forums, online reviews), personal SMS text mes- sage archives can be valuable information sources in their own right, both for the people who contributed to their cre- ation and for others (e.g., historians or researchers) who may subsequently gain authorized access to such collec- tions. Test collections have been built for a number of informa- tion retrieval tasks, both in shared-task venues such as the Text Retrieval Conference (TREC) and for more focused development. We are, however, not aware of any prior work on characterizing the effectiveness of searching SMS con- tent. As part of the DARPA Broad Operational Language Translation (BOLT) program, the Linguistic Data Consor- tium (LDC) released some SMS corpora with the principal goal of supporting research on machine translation for in- formally written content (Song and others, 2014). In this paper, we describe our development of an information re- trieval test collection based on those LDC SMS corpora. The paper is organized as follows: the next section reviews some related work, the data collection is described in sec- tion 3, section 4 describes topics used in the test collection, section 5 gives the relevance assessment procedure and we conclude in section 6. 2. Related Work Although formal dissemination-oriented content such as news stories and scientific papers have been the focus of much information retrieval research, there has also been some work on development of test collections for more in- formal user-generated content in which there is potentially some interaction among the content creators. Among these are the CLEF Cross-Language Speech Retrieval Track fo- cused on interviews (in which interviewer and interviewee co-construct the interview, passages from which are to be found) (Pecina and others, 2007), the TREC Microblog track (focused on Twitter content, some of which is directed between specific users) (Lin and others, 2014), the TREC Legal Track (focused on email content, some of which is easily threaded into conversations) (Grossman and others, 2011), and the the TREC Blog Track (some of which in- volves comments on other …

Content truncated to 500 words