Handling Big Data and Sensitive Data Using EUDAT’s Generic Execution Framework and the WebLicht Workflow Engine.

Handling Big Data and Sensitive Data Using EUDAT’s Generic Execution Framework and the WebLicht Workflow Engine. Claus Zinn, Wei Qui, Marie Hinrichs, Emanuel Dima, and Alexandr Chernov University of T¨ubingen Wilhelmstrasse 19, 72074 T¨ubingen, Germany {claus.zinn, wei.qui, marie.hinrichs, emanuel.dima, alexandr.chernov}@sfs.uni-tuebingen.de Abstract Web-based tools and workflow engines can often not be applied to data with restrictive property rights and to big data. In both cases, it is better to move the tools to the data rather than having the data travel to the tools. In this paper, we report on the progress to bring together the CLARIN-based WebLicht workflow engine with the EUDAT-based Generic Execution Framework to address this issue. Keywords: Generic Execution Framework, WebLicht, Big Data, Sensitive Data 1. Introduction Our work addresses two challenges that affect the appli- cability of workflows for some data sets. First, restrictive property rights may forbid research data to leave their home institution, and therefore, data transferal to other institu- tions for processing is not allowed. The second issue con- cerns the size of the data, with big data often causing pro- hibitive overhead once it is necessary to send such data back and forth to the various tools of a scientific tool pipeline. In both cases, it is desirable to bring the workflow engine to the data, rather than having the data travel to the tools. The EUDAT project is currently developing the Generic Execution Framework (GEF). The GEF aims at providing a framework that allows the execution of scientific workflows in a computing environment close to the data. In this paper, we describe how WebLicht needs to be adapted to render its services compatible with the GEF. The remainder of the paper is structured as follows. In Sect. 2, we give some technical background on the Generic Execution Framework and WebLicht. Sect. 3 specifies the GEF-WebLicht integration from WebLicht’s side. This in- cludes user-centric front-end considerations as well as nec- essary adaptations to WebLicht’s back-end and the process- ing services connected to WebLicht, which need to be con- verted to GEF compatible web services. An integral part is a translation mechanism that liaises the WebLicht workflow engine with the GEF environment, and which ensures that all data transfers occur at the GEF’s host institution, which also hosts all data. Sect. 4 reviews the service compatibility of WebLicht from the point of view of a GEF administra- tor, and in Sect. 5, we reconsider WebLicht’s perspective. In Sect. 6, we conclude. 2. Background 2.1. The Generic Execution Framework One underlying motivation for the GEF is that datasets have become much larger than the tools that process them. It is thus more efficient to move the tools to the data rather than the data to the tools, given that the tools do not require a large amount of processing power. This argument is often supported by restrictive property or privacy rights attached to the data; here data transfer to tools not under control by the property holder is often not possible. Also, the transfer …

Content truncated to 500 words