LexiDB: Patterns & Methods for Corpus Linguistic Database Management

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 3128–3135 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 3128 LexiDB: Patterns & Methods for Corpus Linguistic Database Management Matthew Coole, Paul Rayson, John Mariani Lancaster University Lancaster, UK {m.coole, p.rayson, j.mariani}@lancaster.ac.uk Abstract LexiDB is a tool for storing, managing and querying corpus data. In contrast to other database management systems (DBMSs), it is designed specifically for text corpora. It improves on other corpus management systems (CMSs) because data can be added and deleted from corpora on the fly with the ability to add live data to existing corpora. LexiDB sits between these two categories of DBMSs and CMSs, more specialised to language data than a general-purpose DBMS but more flexible than a traditional static corpus management system. Previous work has demonstrated the scalability of LexiDB in response to the growing need to be able to scale out for ever-growing corpus datasets. Here, we present the patterns and methods developed in LexiDB for storage, retrieval and querying of multi-level annotated corpus data. These techniques are evaluated and compared to an existing CMS (Corpus Workbench CWB - CQP) and indexer (Lucene). We find that LexiDB consistently outperforms existing tools for corpus queries. This is particularly apparent with large corpora and when handling queries with large result sets. Keywords: corpus, database 1. Introduction Recent years have seen a development of many new inno- vative DBMSs (Database Management Systems) that have moved away from the traditional relational database model. The No-SQL movement, in particular, has concerned it- self largely with a system’s ability to scale out to handle increasingly large datasets. Language data in the worlds of corpus and computational linguistics, digital humanities and lexicography has also seen the scale of what is consid- ered a ‘large’ corpus grow from one million words in LOB (Johansson et al., 1986), and one hundred million words British National Corpus (Leech, 1992) to billions and tens of billions of words TenTen (Jakubíček et al., 2013), COW (Schäfer and Bildhauer, 2012), Hansard1, EEBO-TCP2 over the last forty years. However, specialist corpus storage and retrieval tools have lagged behind the developments made by more generic DBMSs. In addition, modern DBMSs do not provide the necessary query functionality to satisfy the types of queries used by corpus linguists. Moreover, ex- isting database models do not provide efficient storage and retrieval for consecutive words in running text, sequential queries and contextual sorting that are central to the require- ments for textual corpora. These issues present a problem for linguists, humanists and others wishing to work with ex- treme scale corpus datasets using the corpus methodology. Conventional solutions to storage and querying of corpus datasets (such as relational DBMSs and index tools such as Lucene3) have worked in the past for smaller-scale datasets. However, they have always relied upon an application layer built on top of a DBMS to translate specialist corpus queries into a query for a more generic data model such …

Content truncated to 500 words