Features for Generic Corpus Querying

Features for Generic Corpus Querying Thomas Eckart, Christoph Kuras, Uwe Quasthoff Natural Language Processing Group, University of Leipzig Augustusplatz 10, 04109 Leipzig, Germany Email: {teckart, ckuras, quasthoff}@informatik.uni-leipzig.de Abstract The availability of large corpora for more and more languages enforces generic querying and standard interfaces. This development is especially relevant in the context of integrated research environments like CLARIN or DARIAH. The paper focuses on several applications and implementation details on the basis of a unified corpus format, a unique POS tag set, and prepared data for word similarities. All described data or applications are already or will be in the near future accessible via well-documented RESTful Web services. The target group are all kinds of interested persons with varying level of experience in programming or corpus query languages. Keywords: corpus, corpus query, Web services 1. Introduction A corpus is called generic if it is designed to fit for multiple purposes. This usually implies a comparably large size, high quality and a certain kind of added value like annotation or additional pre-calculated data. The Leipzig Corpora Collection provides generic corpora for more than 200 languages. In 2015 enhanced functionality were developed and new features were added to simplify the usage of the LCC data as a generic source for annotated text material. The data can be queried using a newly developed Web frontend1 or directly via Web services2. Some of the feature values described below depend on the quality of the corpus. For instance, duplicates (and near duplicates) in the corpus generate unnatural significant word co-occurrences (see sect. 2.3). So, only extensive preprocessing with emphasis on corpus quality will generate reliable data with general acceptance. 2. Feature Overview The following list of features can be considered as a standard set of corpus queries. These features have a different range of application like general linguistics, lexicography, information extraction, and ontology learning. Recipients are either human users or other algorithms. The features are designed to have quick response time which allows many queries. Moreover, they should serve multiple interests and the output should be usable for all kinds of recipients. 1 Available at http://corpora.informatik.uni-leipzig.de/ 2 http://wortschatzwebservices.informatik.uni- leipzig.de 2.1. Word frequency and distribution The following information is available for all words in a corpus and also for a pre-selected list of multiword units (MWUs). For a certain language, this list of MWUs contains all article titles from Wikipedia in this language and possibly more data like additional person names, technical terms and phraseology. In the following, the term word is also used for these MWUs. The word frequency is available not only for the whole corpus. In the case of news corpora where a corpus production is performed on a yearly basis there is a frequency distribution over time for every word. Moreover, there is a distribution w.r.t. sources which shows whether a word is of general use or belongs to a subset like a subject area 2.2. Part-of-Speech Tags A set of different POS-taggers is used to provide POS-tags for as many languages …

Content truncated to 500 words