Fast Syntactic Searching in Very Large Corpora for Many Languages

Fast syntactic searching in very large corpora for many languages Miloš Jakubíˇcek‡†, Adam Kilgarriff†, Diana McCarthy† , Pavel Rychlý‡† {xjakub, pary}@fi.muni.cz adam@lexmasterclass.com, diana@dianamccarthy.co.uk ‡Masaryk University, Brno, Czech Republic †Lexical Computing, Ltd., Brighton, United Kingdom Abstract. For many linguistic investigations, the first step is to find examples. In the 21st century, they should all be found, not invented. Thus linguists need flexible tools for finding even quite rare phenomena. To support linguists well, they need to be fast even where corpora are very large and queries are complex. We present extensions to the CQL ’Corpus Query Language’ for intuitive creation of syntactically rich queries, and demonstrate that they can be computed quickly within our tool even on multi-billion word corpora. Keywords: corpus search, large corpora, CQL, syntactic search 1 Introduction It is important for much linguistic research to find datasets relating to hunches and hypotheses. Linguists and system developers would like to be able to find large numbers of examples quickly and easily. Our tool computes the Corpus Query Language (CQL) (Christ and Schulze, 1994) queries rapidly on large corpora using the Manatee corpus manager system as its backend which is based on stream processing techniques and has been described in (Rychlý, 2000). We have large (more than 100 million-word) corpora loaded into the tool and accessible over the web for 25 languages and the number rises month by month (Reddy et al., 2010). For English, Italian and German we have very large (more than 1.5 billion-word) corpora, and for English, also a corpus of 5.5 billion words (Pomikálek et al., 2009). We have recently augmented the CQL formalism with extensions to the ‘within’ operator and a new ‘containing’ operator, to better support queries relating to phrase structure. In this paper we first summarise standard CQL, then describe the extensions to the ‘within’ operator, then introduce the ‘containing’ operator, and finally indicate computation times for sample queries. For the same purpose, the ‘meet’ and ‘union’ operators have been extended so that they can be used at any token position in the query as we describe in this paper. For the bulk of the paper we assume a corpus which is lemmatised and part-of-speech tagged (as these processes are fast and available for many languages, and have been applied to most of our corpora) but not parsed. 2 Related Work There are numerous other corpus query tools available. Here we give brief references to several that are either widely used on large corpora or which concentrate on syntactic search. • the IMS Stuttgart CorpusWorkBench (Christ and Schulze, 1994), limited to 2 billion-words corpora, • a system presented in (Davies, 2009) which uses standard relational database technology, not closely related to syntactic search, • tools being part of the Tigersearch project (König and Lezius, 2000), currently unmaintained, • the Gsearch corpus query system (Steffan et al., 2001), PACLIC 24 Proceedings 741 3 Core CQL A CQL query is a pattern which may match a token or series of tokens in the corpus. Each …

Content truncated to 500 words