TEITOK: Text-Faithful Annotated Corpora

TEITOK: Text-Faithful Annotated Corpora Maarten Janssen CLUL, Universidade de Lisboa Lisboa, Portugal Maarten.Janssen@campus.ul.pt Abstract TEITOK is a web-based framework for corpus creation, annotation, and distribution, that combines textual and linguistic annotation within a single TEI based XML document. TEITOK provides several built-in NLP tools to automatically (pre)process texts, and is highly customizable. It features multiple orthographic transcription layers, and a wide range of user-defined token-based annotations. For searching, TEITOK interfaces with a local CQP server. TEITOK can handle various types of additional resources including Facsimile images and linked audio files, making it possible to have a combined written/spoken corpus. It also has additional modules for PSDX syntactic annotation and several types of stand-off annotation. Keywords: annotated corpora, document transcription, TEI 1. Introduction Corpora that are based on manuscripts typically come in two flavours. On the one hand there are those created by philologists, which focus on faithfully representing the con- tent of the manuscript and include line breaks, typesetting information (colour, italics), changes of hand, deleted frag- ments, etc. And on the other hand there are those cre- ated by (corpus) linguists, which capture linguistic meta- information including POS tags, lemmas, normalized or- thography, semantic classifications, grammatical parses, etc. There are few corpora that include both, which can be at- tributed largely to three factors. Firstly, the two types of corpora are created by different audiences that often don’t see the value of the other type of information: philologists see little reason for including POS tags, while corpus lin- guists tend to view typesetting information as ephemeral. Secondly, both types of corpora are labour-intensive, and combining both types of information makes it even more so. And thirdly, there are no or hardly any tools that facili- tate the creation of such combined corpora. The first point is mostly due to a mutual misunderstand- ing. On the one hand, using linguistic annotations without access to the textual information can often lead to conclu- sion that have nothing to do with the actual manuscript, but more with the choices by the transcriber: manuscript are often hard to read and degraded, and transcription of- ten involves reconstructions and interpretations. And from the other side, linguistic annotations are not merely use- ful for grammatical or statistical analysis, but also provide richer search options on the manuscript transcriptions, for instance by searching lemma rather than (often deviant) or- thography. The fact that a combined corpus is labour intensive is very true, but existing linguistically annotated manuscript-based corpora are almost always based on prior (textually anno- tated) transcriptions. Most linguistically annotated corpora do not keep the textual annotation, however, due to the lack of appropriate tools. TEITOK is an online tool that aims to provide a solution to that: a tool that allows adding layers of linguistic annotations to textually annotated texts. This results in a corpus combining both types of information, which adds value in all those contexts where there is ad- ditional information in the source document or where easy reading is advantageous. This …

Content truncated to 500 words