corpus-tools.org: An Interoperable Generic Software Tool Set for Multi-layer Linguistic Corpora

corpus-tools.org: An Interoperable Generic Software Tool Set for Multi-layer Linguistic Corpora Stephan Druskat1,2, Volker Gast1, Thomas Krause2, Florian Zipser2 1Friedrich Schiller University, Dept. of English and American Studies, Ernst-Abbe-Platz 8, D-07743 Jena 2Humboldt-Universit¨at zu Berlin, Dept. of German Studies and Linguistics, Unter den Linden 6, D-10099 Berlin {stephan.druskat, volker.gast}@uni-jena.de, krauseto@hu-berlin.de, f.zipser@gmx.de Abstract This paper introduces an open source, interoperable generic software tool set catering for the entire workflow of creation, migration, annotation, query and analysis of multi-layer linguistic corpora. It consists of four components: Salt, a graph-based meta model and API for linguistic data, the common data model for the rest of the tool set; Pepper, a conversion tool and platform for linguistic data that can be used to convert many different linguistic formats into each other; Atomic, an extensible, platform-independent multi-layer desktop annotation software for linguistic corpora; ANNIS, a search and visualization architecture for multi-layer linguistic corpora with many different visualizations and a powerful native query language. The set was designed to solve the following issues in a multi-layer corpus workflow: Lossless data transition between tools through a common data model generic enough to allow for a potentially unlimited number of different types of annotation, conversion capabilities for different linguistic formats to cater for the processing of data from different sources and/or with existing annotations, a high level of extensibility to enhance the sustainability of the whole tool set, analysis capabilities encompassing corpus and annotation query alongside multi-faceted visualizations of all annotation layers. Keywords: corpus tools, multi-layer corpora, interoperability 1. Introduction This paper introduces an interoperable generic software tool set1 which caters for the entire workflow of creation, migration, annotation, query and analysis of multi-layer linguistic corpora. The tool set consists of four major com- ponents: 1. Salt (Zipser and Romary, 2010) (Humboldt- Universit¨at zu Berlin, 2016b), a meta model and API for linguistic data, working on a generic graph structure and functioning as a common data model for the other tools; 2. Pepper (Zipser et al., 2011) (Humboldt-Universit¨at zu Berlin, 2016a), a conversion tool and platform for lin- guistic data that can be used to convert a large number of different linguistic formats into each other; 3. Atomic (Druskat et al., 2014) (Friedrich Schiller University Jena, 2015), an extensible, platform- independent multi-layer desktop annotation software for linguistic corpora; 4. ANNIS (Krause and Zeldes, 2014) (Humboldt- Universit¨at zu Berlin, 2015), a search and visual- ization architecture for multi-layer linguistic corpora with many different visualizations and a powerful na- tive query language. All components are open source under the permissive Apache License, Version 2.02. We define the corpus-tools.org tools as a software set rather than a stack or a toolchain, as they are designed for close 1http://corpus-tools.org, (corpus-tools.org Develop- ment Team, 2015). 2www.apache.org/licenses/LICENSE-2.0.html. integration, but can be used independently of each other and must be differentiated by their target functions: Salt and Pepper are primarily infrastructure tools, i.e., they pro- vide infrastructural foundations for computer-based corpus research; Atomic and ANNIS are primarily workflow tools, i.e., they contribute to the actual …

Content truncated to 500 words