MULTEXT: Multilingual Text Tools and Corpora

MULTEXT : Multilingual Text Tools and Corpora Nancy Ide and Jean V6ronis LABORATOIRE PAROLE ET LANGAGE CNRS & Universitd de Provence 29, Avenue Robert Schuman 13621 Aix-en-Provence Cedex 1 (France) e-mail: ide@fraixll .univ-aix. fr, veronis@fraixll .univ-aix. fr Abstract, MULTEXT (Multilingual Text "Fools and Corpora) is the largest project funded in the Commission of European Communities Linguistic Research and Engineering Program. The project will contribute to the development of generally usable software tools to manipulate and analyse text corpora and to create multi- lingual text corpora with structural and linguistic markup. It will attempt to establish conventions for the encoding of such corpora, building on and contributing to the preliminary recommendations of the relevant international and European standardization initiatives. MULTEXT will also work towards establishing a set of guidelines for text software development, which will be widely published in order to enable future development by others. All tools and data developed within the project will be made freely and publicly available. Keywords. multi-lingual corpora, text markup, text software, corpus annotation. 1. Introduction Text-oriented methods and software tools have come to be of primary interest to the NLP community. However, existing tools for natural language processing (NLP) and machine translation (MT) corpus-based research are typically embedded in large, non-adaptable systems which are fundamentally incompatible. Little effort has been made to develop software standards, and software reusability is virtually non-existent. As a result, there is a serious lack of generally usable tools to manipulate and analyze text corpora that arc widely available for research, especially for multi-lingual al)plications. At the same time, the availability of data is hampered by a lack of well-established standards for encoding corpora. Although the Text Encoding Initiativc (TEI) has provided guidelines for text encoding [Sper94], they arc so far largely untested on real-scale data, especially multi-lingual data. Further, the TEl Guidelines offer a broad range of text encoding solutions serving a w~riety of disciplines and applications, and are not intended to provide specific guidance for the purposes of NLP and MT corpus-based research. MIJLTEXT (Multilingual Text Tools and Corpora) ix a recently initiated large-scale project funded under tim Commission of European Communities Linguistic Research and Engineering Program, which is intended to address these problems. The project will contribute to the development of generally usable software tools to nmnipulate and analyse text corpora and to create multi- lingual text corpora with structural and linguistic markt, p. It will attempt to establish conventions for the encoding of such corpora, building on and contributing to the preliminary recommendations of the relevant international and European standardization initiatives. MULTEXT will also work towards establishing a set of guidelines for text software development, which will be wklely published in order to enable future development by others. The project consortimn, consisting of eight academic and research institutions and six major European industrial partners, is committed to make its results, namely corpus, related tools, specifications and accompanying documentation, freely and publicly available. 2. Project Overview At the outset of the project, the consortimn will undertake to …

Content truncated to 500 words