Querying and Updating Treebanks: A Critical Survey and Requirements Analysis

Querying and Updating Treebanks: A Critical Survey and Requirements Analysis Catherine Lai and Steven Bird Department of Computer Science and Software Engineering University of Melbourne, Victoria 3010, Australia {clai,sb}@csse.unimelb.edu.au Abstract Language technology makes extensive use of hierarchi- cally annotated text and speech data. These databases are stored in flat files and manipulated using corpus-specific query tools or special-purpose scripts. While the size of these databases and the range of applications has grown rapidly in recent years, neither method for managing the data has led to reusable, scalable software. The formal properties of the query languages are not well under- stood. Hence established methods for indexing tree data and optimizing tree queries cannot be employed. We analyze a range of existing linguistic query languages, and adduce a set of requirements for a reusable, scalable linguistic query language. 1 Introduction Corpora form the backbone of language technol- ogy. However corpora usually need to be annotated with structural information describing, for example, syntax or phonology. Query languages are neces- sary to extract useful information from these mas- sive data sets. Moreover, annotated corpora require thousands of hours of manual annotation to create, revise and maintain. Query languages are also use- ful during this process. For example, queries can be used to find parse errors or to transform annotations into different schemes. However, the query languages currently available for annotated corpora suffer from several problems. Firstly, updates are not supported as query languages focus on the needs of linguists searching for syntactic constructions. Secondly, their relationship to existing database query languages is poorly understood, making it difficult to apply standard database indexing and query optimization techniques. As a consequence they do not scale well. Finally, linguistic annotations have both a sequential and a hierarchical organization. Query languages must support queries that refer to both of these types of structure simultaneously. Such hybrid queries should have a concise syntax. The interplay between these factors has resulted in a variety of mutually-inconsistent approaches. This paper aims to describe the query language requirements for navigating and modifying struc- turally annotated corpora. We focus on tree struc- tured annotation. This entails sequential structure. This paper is organized as follows. Section 2 surveys six linguistic tree querying languages and where appropriate the data models they are based on. The survey is the basis of the linguistic tree query requirements presented in Section 3. We con- clude in Section 4 with suggested areas for further work. 2 Tree Models and Query Languages The prototypical hierarchical linguistic annotation is the syntax tree, an ordered tree where termi- nals (leaves) contain the text of a sentence being analyzed. Non-terminals represent syntactic cate- gories, and the hierarchical organization represents constituency. The leaf level is usually considered immutable. The queries in Figure 1 have been cho- sen to highlight the expressive capabilities of cur- rent tree query languages. A query language must be able to accurately specify which subtrees to match in a corpus. This means quantifying the existence of nodes and succintly stating …

Content truncated to 500 words