A Document Repository for Social Media and Speech Conversations

A Document Repository for Social Media and Speech Conversations Adam Funk, Robert Gaizauskas, Benoit Favre University of Sheffield, University of Sheffield, Aix Marseille Universit´e Sheffield (UK), Sheffield (UK), Marseille (France) a.funk@sheffield.ac.uk, r.gaizauskas@sheffield.ac.uk, benoit.favre@lif.univ-mrs.fr Abstract We present a successfully implemented document repository REST service for flexible SCRUD (search, create, read, update, delete) storage of social media and speech conversations, using a GATE/TIPSTER-like document object model and providing a query language for document features. This software is currently being used in the SENSEI research project and will be published as open-source software before the project ends. It is, to the best of our knowledge, the first freely available, general purpose data repository to support large-scale multimodal (i.e., speech or text) conversation analytics. Keywords: Document repository; social media; REST service 1. Introduction Conversational interaction, including social media stream- ing and spoken conversation, is a natural paradigm for in- teractions with customers and other users. In the SENSEI project we are going beyond keyword search and sentence- based analysis of such conversations to provide automatic descriptions and summaries of who said what and of opin- ions held on relevant subjects. To this end, we need to store structured data for millions of constituents of the conversa- tions, including the original texts and associated metadata as well as annotations generated in the course of syntactic, semantic, and pragmatic analysis, summarization, cluster- ing, etc. We have therefore designed a custom repository for this purpose, which we will publish as open-source soft- ware as part of the project’s dissemination and exploitation activity. It is, to the best of our knowledge, the first freely available, general purpose data repository to support large- scale multimodal (i.e., speech or text) conversation analyt- ics. Our conversational repository offers SCRUD (search, cre- ate, read, update, delete) functionality (Martin, 1983) over richly annotated documents. It runs as a standard REST service with JSON for data interchange. This offers a num- ber of advantages for easy interaction, testing, and debug- ging, and JSON avoids the verbosity and high-level parsing hassles associated with XML (e.g., walking a DOM tree or constructing one with SAX); it is also trivially easy to process with Python’s standard library and fairly easy to process in Java with the Jackson library (MongoLab, 2011; Saloranta, 2013; Ecma International, 2013). 2. Document model The main unit of data storage in the repository is a docu- ment object similar to the GATE or TIPSTER model (Wilks et al., 2000; Cunningham et al., 1997), which allows arbit- rary data to be stored as stand-off annotations (grouped in named sets), annotation features, and document-wide fea- tures. The document is represented in the repository input and output as a JSON object1 containing an unique integer 1The JSON object corresponds to other languages’ map, dic- tionary, or associative array. (Ecma International, 2013) id assigned by the repository, an optional name string, a content string representing the plain text content of the document (e.g., of a web page with the HTML tags stripped), a features object, and an …

Content truncated to 500 words