KonText: Advanced and Flexible Corpus Query Interface

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 7003–7008 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 7003 KonText: Advanced and Flexible Corpus Query Interface Tom´aˇs Mach´alek Institute of the Czech National Corpus, Faculty of Arts - Charles University n´am. Jana Palacha 1/2, 116 38 Praha 1, Czech Republic tomas.machalek@ff.cuni.cz Abstract We present an advanced, highly customizable corpus query interface KonText built on top of core libraries of the open-source corpus search engine NoSketch Engine (NoSkE). The aim is to overcome some limitations of the original NoSkE user interface and provide integration capabilities allowing connection of the basic search service with other language resources (LRs). The introduced features are based on long-term feedback given by the users and researchers of the Czech National Corpus (CNC) along with other LRs providers running KonText as a part of their services. KonText is a fully operational and mature software deployed at the CNC since 2014 that currently handles thousands user queries per day. Keywords: corpus query interface, open-source, web application, distributed application, corpus linguistics 1. General overview and motivation Creating a fully featured corpus search engine with a pow- erful query language, analytical functions, user-friendly in- terface, performance scalable to billion tokens corpora and tens or more concurrent users is a tremendous task which takes many years to accomplish. Only a limited number of projects have been able to reach production-ready sta- tus and general awareness in linguistic community. Among them, Corpus Workbench (Hardie, 2012), SketchEngine (Kilgarriff et al., 2014), BlackLab (Does et al., 2017) or Ko- rAP (Diewald and Margaretha, 2017) should be mentioned. Looking for a suitable corpus search system, our experi- ence shows that there is simply not a clear winner espe- cially when considering both supported features and scal- ability (in terms of corpus size and number of users). A good example of this performance-feature dilemma can be seen on two systems we have experience with - Annis (Krause and Zeldes, 2016) and NoSketch Engine (NoSkE) (Rychl´y, 2007). While the former system provides rich, multi-layered annotation capabilities at the cost of limited usability for large corpora (Krause, 2019), the latter system is able to handle billion token corpora reasonably fast but only for simple, single-layer token and span annotations. Instead of trying to build another search engine fully com- pliant with our needs, we took a pragmatic approach by choosing a simple, scalable search engine and building a new interface and auxiliary services integrating the search engine with other data resources and providing many fea- tures missing in the original software. KonText builds on our long-term experience with NoSkE which is an open-source version of a commercial corpus search software Sketch Engine (SkE) by Lexical Comput- ing Ltd. During our initial attempts to enhance the orig- inal user interface (Mach´alek and Kˇren, 2013), we were faced by increasing number of requested features. As we also had a different approach to the application architecture, we have gradually rewritten the original …

Content truncated to 500 words