Two Architectures for Parallel Processing of Huge Amounts of Text

Two Architectures for Parallel Processing of Huge Amounts of Text Mathijs Kattenberg,♦ Zuhaitz Beloki,♠ Aitor Soroa,♠ Xabier Artola,♠ Antske Fokkens,♣ Paul Huygen♣ and Kees Verstoep♣ ♦SURFsara, Science Park 140, 1098 XG Amsterdam ♠IXA NLP Group, University of the Basque county, 649 Posta kutxa, 20080 Donostia ♣The Network Institute, Vrije Universiteit, Boelelaan 1105, 1081 HV Amsterdam, mathijs.kattenberg@surfsara.nl, {zuhaitz.beloki,a.soroa,xabier.artola}@ehu.es {antske.fokkens,p.e.m.huijgen,c.verstoep}@vu.nl Abstract This paper presents two alternative NLP architectures to analyze massive amounts of documents, using parallel processing. The two architectures focus on different processing scenarios, namely batch-processing and streaming processing. The batch-processing scenario aims at optimizing the overall throughput of the system, i.e., minimizing the overall time spent on processing all documents. The streaming architecture aims to minimize the time to process real-time incoming documents and is therefore especially suitable for live feeds. The paper presents experiments with both architectures, and reports the overall gain when they are used for batch as well as for streaming processing. All the software described in the paper is publicly available under free licenses. Keywords: Parallel processing, big data, system architecture 1. Introduction The amount of textual information available to us to- day is enormous and is increasing by the day. Current technologies can analyze these texts at a very detailed level, but NLP analyses are time-consuming. We dis- tinguish two issues that need to be dealt with when monitoring textual data that is both abundant and con- stantly updated. First, the problem of dealing with large amounts of data. Second, the problem of being able to respond immediately when new information comes in. This paper introduces two architectures that make use of parallel processing to address these issues. Depend- ing on the requirements and quantity of the data this can be done in different ways. Extremely large quan- tities of data that do not require instant analysis can be processed following a batch paradigm, where an en- tire batch of documents is processed in one go. How- ever, if immediate updates of new information is re- quired, focus lies on a quick turn-out per document. Streaming computing (Cherniack et al., 2003) expects documents to arrive at any moment, and aims at re- ducing the elapsed time needed to process one sin- gle document. The two architectures we present each tackle one of these scenarios. The batch processing ar- chitecture provides a general setup for NLP pipelines that can be used for processing large quantities of documents distributing jobs over various nodes. The streaming architecture can deal with live feeds using parallellization to optimize processing time per docu- ment. We illustrate the idea behind the architectures through the NewsReader1 use case, where linguistic analyses are used to form a “history recorder” of the news pub- lished around a certain topic. This paper is structured as follows. In Section 2., we describe the background of this work. This is followed by a presentation of the NLP modules we use in Section 3. Sections 4. and 5. introduce the Hadoop architecture for batch processing and the Storm architecture …

Content truncated to 500 words