KorAP Architecture ― Diving in the Deep Sea of Corpus Data

KorAP Architecture – Diving in the Deep Sea of Corpus Data Nils Diewald1, Michael Hanl1, Eliza Margaretha1, Joachim Bingel2, Marc Kupietz1, Piotr Ba´nski1, Andreas Witt1,3 1 Institut für Deutsche Sprache, Mannheim {diewald,hanl,margaretha,kupietz,banski,witt}@ids-mannheim.de 2 Centre for Language Technology, University of Copenhagen bingel@hum.ku.dk 3 Universität Heidelberg Abstract KorAP is a corpus search and analysis platform, developed at the Institute for the German Language (IDS). It supports very large corpora with multiple annotation layers, multiple query languages, and complex licensing scenarios. KorAP’s design aims to be scalable, flexible, and sustainable to serve the German Reference Corpus DEREKO for at least the next decade. To meet these requirements, we have adopted a highly modular microservice-based architecture. This paper outlines our approach: An architecture consisting of small components that are easy to extend, replace, and maintain. The components include a search backend, a user and corpus license management system, and a web-based user frontend. We also describe a general corpus query protocol used by all microservices for internal communications. KorAP is open source, licensed under BSD-2, and available on GitHub. Keywords: KorAP, microservices, large corpus data 1. Introduction With DEREKO, the largest linguistically motivated corpus of contemporary German, growing beyond 25 billion words (Kupietz and Lüngen, 2014), new challenges for corpus management have arisen at the Institute for the German Language (IDS),1 especially as the textual data is enriched by several layers of annotations, multiplying the quantity of data immensely. A platform capable of handling such large amounts of data, while providing a wide variety of features for searching, analysing, and legally protecting the corpus, needs an architecture focussing on scalability, flex- ibility, and sustainability. In this paper, we present our approach to meet these re- quirements: a microservice-based architecture. KorAP2 (“Korpusanalyseplattform der nächsten Genera- tion”; Ba´nski et al., 2013) is a web-based corpus analysis platform, consisting of small independent components that are easy to extend, to replace, and to maintain. All components communicate through KoralQuery, a gen- eral corpus query protocol. As we design the communica- tion protocol to be easily extensible as well as backward compatible, our approach also addresses the fact that soft- ware may change or needs to be replaced over time. In this manner, we strive to ensure sustainability and flexibility in the maintenance of the system for the next 15-20 years, thus targeting the life span of COSMAS II (Bodmer, 1996), the corpus analysis platform that KorAP is designed to replace. The present contribution outlines the architecture of KorAP, describes the necessary building blocks, and discusses the capabilities and caveats of searching, analysing and legally protecting corpus data against the requirements of a large annotated text corpus such as DEREKO. 1http://ids-mannheim.de/ 2http://korap.ids-mannheim.de/ 2. KorAP Infrastructure Requirements typically associated with scalable, flexible, and sustainable systems were the most important crite- ria for architectural decisions in KorAP. These priorities led to a microservice-based design, “an approach to de- veloping a single application as a suite of small services, each running in its own process and communicating with lightweight mechanisms” (Lewis and …

Content truncated to 500 words