The TXM Platform: Building Open-Source Textual Analysis Software Compatible with the TEI Encoding Scheme

The TXM Platform: Building Open-Source Textual Analysis Software Compatible with the TEI Encoding Scheme* Serge Heiden ICAR Laboratory, University of Lyon, 15 parvis René Descartes - ENS de Lyon, 69342 Lyon, France slh@ens-lyon.fr Abstract. This paper describes the rationale and design of an XML-TEI encoded corpora compatible analysis platform for text mining called TXM. The design of this platform is based on a synthesis of the best available algorithms in existing textometry software. It also relies on identifying the most relevant open-source technologies for processing textual resources encoded in XML and Unicode, for efficient full-text search on annotated corpora and for statistical data analysis. The architecture is based on a Java toolbox articulating a full-text search engine component with a statistical computing environment and with an original import environment able to process a large variety of data sources, including XML-TEI, and to apply embedded NLP tools to them. The platform is distributed as an open-source Eclipse project for developers and in the form of two demonstrator applications for end users: a standard application to install on a workstation and an online web application framework. Keywords: xml-tei corpora, search engine, statistical analysis, textometry, open-source. * The work reported in this paper was supported by French ANR grant #ANR-06-CORP-029. Copyright 2010 by Serge Heiden 1 Introduction Textometry is a textual data analysis methodology born in France in the 80s in relation with the data analysis framework designed by (Benzecri et al., 1973a,b). A first synthesis of the methodology can be found in (Lebart et al, 1997). It somewhat differentiates from text mining by the fact that it tries to always combine various statistical analysis techniques, like factorial correspondence analysis or hierarchical ascendant classification, with full-text search techniques like kwic concordances to be able to always get back to the precise original editorial context of any textual event participating to the analysis. It tries to always relate golden nuggets found in corpora to the context of their original data source. Thus it involves more an interaction with corpora than a distillation process of them. PACLIC 24 Proceedings 389 In 2007 a project joining textometry teams from four French universities1 started a four year project to build an original open-source software framework for a new generation of application software for textometry2. The main goals of the project are: • synthesize the best available algorithms for textometry analysis with the most up to date annotated corpora models • implement them with the best available and sustained open-source components • be compatible with Unicode (2006), XML and TEI (2008) standard encoded data sources • be able to efficiently analyze corpora of several hundred million tagged words • be compatible with usual NLP software (like taggers and lemmatizers) • distribute a framework toolbox for developers to build new applications • demonstrate a Windows and Linux prototype application for the end researcher users from the humanities and social sciences • demonstrate an equivalent web based client/server prototype application • package the applications for easy install and deployment • …

Content truncated to 500 words