Encoding standards for large text resources: The Text Encoding Initiative

Encoding standards for large text resources: The Text Encoding Initiative Nancy Ide Department of Computer Science Vassar College Poughkeepsie, New York 12601 (U.S.A.) LaboratoirE Parole et Langage CNRS/Universitd de Provence 29, Avenue R.obert Schuman 13621 Aix-en-Provence Cedex 1 (France) e-maih ide@cs, vassar, edu Abstract. The Text Encoding Initiative (TEl) is an international project established in 1988 to develop guidelines for the preparation and interchange of electronic texts for research, and to satisfy a broad range of uses by the language industries more generally. The need for standardized encoding practices has become inxreasingly critical as the need to use and, most importantly, reuse vast amounts of electronic text has dramatically increased for both research and industry, in particular for natural language processing. In January 1994, the TEl isstled its Guidelines for the Fmcoding and hiterehange of Machine-Readable Texts, which provide standardized encoding conventions for a large range of text types and features relevant for a broad range of applications. Keywords. Encoding, markup, large text resources, corpora, SGML. 1. Introduction The past few years have seen a burst of activity in the development of statistical methods which, applied to massive text data, have in turn enabled the dcvelopnmnt of increasingly comprehensive and robust models of language structure and use. Such models are increasingly recognized as an inwduable resource for natural langu:lge processing (NLP) tasks, inch,ding machine translation. The upsurge of interest in empricial methods for language modelling has led inevitably to a need for massive collections of texts of all kinds, including text collections which span genre, register, spoken and written data, etc., as well as domain- or application- specific collections, and, especially, multi-lingual collections with parallel translations. In tile latter half of the 1980's, very few appropriate or adequately large text collections existed for use in computational linguistics research, especially for languages other than English. Consequently, several efforts to collect and disseminate large mono- and multi-lingual text collections have been recently established, including the ACL Data Collection Initiative (ACL/DCI), the European Corpus Initiative (ECI), which has developed a multilingual, partially parallel corpus, the U.S. Linguistic Data Consortium (LDC), RELATOR and MULTEXT in EuropE, etc. (see Arm.strong-Warwick, 1993). It is widely recognized that such efforts constitute only a beginning for the necessary data collection and dissemination efforts, and that considerable work to develop adequately large and appropriately constituted textual resources still remains. The demand for extensive reusability of large text collections in turn requires the development of standardized encoding formats for this data. It is no longer realistic to distribute data in ad hoc formats, since the eflbrt and resources required to clean tip and reformat the data for local use is at best costly, and in many cases prohibitive. Because much existing and potentially available data was originally formatted R)r the purposes of printing, the information explicitly represented in the encoding concerns a imrticular physical realization of a text rather than its logical strttcture (which is of greater interest for most NLP applications), and the correspondence between the two is often …

Content truncated to 500 words