Extracting Interlinear Glossed Text from LaTeX Documents

Extracting Interlinear Glossed Text from LATEX Documents Mathias Schenner, Sebastian Nordhoff Language Science Press, Freie Universit¨at Berlin, Habelschwerdter Allee 45, 14195 Berlin mathias.schenner@langsci-press.org, sebastian.nordhoff@langsci-press.org Abstract We present texigt, a command-line tool for the extraction of structured linguistic data from LATEX source documents, and a language resource that has been generated using this tool: a corpus of interlinear glossed text (IGT) extracted from open access books published by Language Science Press. Extracted examples are represented in a simple XML format that is easy to process and can be used to validate certain aspects of interlinear glossed text. The main challenge involved is the parsing of TEX and LATEX documents. We review why this task is impossible in general and how the texhs Haskell library uses a layered architecture and selective early evaluation (expansion) during lexing and parsing in order to provide access to structured representations of LATEX documents at several levels. In particular, its parsing modules generate an abstract syntax tree for LATEX documents after expansion of all user-defined macros and lexer-level commands that serves as an ideal interface for the extraction of interlinear glossed text by texigt. This architecture can easily be adapted to extract other types of linguistic data structures from LATEX source documents. Keywords: Interlinear Glossed Text, LaTeX, Parsing 1. Introduction We present a tool for the extraction of structured linguistic data from LATEX source documents and a language resource that has been generated using this tool: a corpus of inter- linear glossed text (IGT) extracted from open access books published by Language Science Press. Many linguistic research papers and books are written in LATEX, using more or less semantic markup. Our goal is to take advantage of this existing structural information and systematically extract linguistic data structures from prop- erly formatted LATEX documents. This route can comple- ment existing tools for harvesting IGT from PDF docu- ments, in particular ODIN (Lewis and Xia, 2010), which have to cope with much more noise in the source chan- nel resulting from lossy pdf-to-text conversion. However, parsing LATEX comes with its own challenges, as detailed in section 5. In addition to extracting data, the tool we are presenting can also be used for validating LATEX encodings of IGT. For example, it can check whether the number of words and morphemes in the source line match those in the gloss lines or whether the formatting used in gloss lines conforms to certain conventions codified by the Leipzig Glossing Rules (Bickel et al., 2008). 2. Interlinear Glossed Text Interlinear glossed text (IGT) is a commonly used format for describing linguistic data, typically comprised of sev- eral word-aligned lines and a single free translation line. A simple example, taken from Holton and Robinson (2014, p.163), is shown in (1). (1) Ke’e fish pi-ga-ussar. 1PL-3SG-catch ‘We’re catching fish.’ The first line contains a phrase in the original language, with words split into morphemes separated by a dash. The second line provides a word-aligned morpheme-by- morpheme analysis, and the final line gives a free trans- …

Content truncated to 500 words