PDF-to-Text Reanalysis for Linguistic Data Mining

PDF-to-Text Reanalysis for Linguistic Data Mining Michael Wayne Goodman, Ryan Georgi, Fei Xia Linguistics Department University of Washington goodmami@uw.edu, rgeorgi@uw.edu, fxia@uw.edu Abstract Extracting semi-structured text from scientific writing in PDF files is a difficult task that researchers have faced for decades. In the 1990s, this task was largely a computer vision and OCR problem, as PDF files were often the result of scanning printed documents. Today, PDFs have standardized digital typesetting without the need for OCR, but extraction of semi- structured text from these documents remains a nontrivial task. In this paper, we present a system for the reanalysis of glyph-level PDF-extracted text that performs block detection, respacing, and tabular data analysis for the purposes of linguistic data mining. We further present our reanalyzed output format, which attempts to eliminate the extreme verbosity of XML output while leaving important positional information available for downstream processes. Keywords: Low Resource Languages, Interlinear Glossed Text (IGT), Corpus Creation 1. Introduction A great deal of linguistic information exists online in the form of academic publications. ODIN, the Online Database of INterlinear text (Lewis and Xia, 2010) was created upon this premise, a resource which makes available language data for approxi- mately 1,500 languages, including linguistic glosses and resource-rich language translations (Lewis et al., 2015). The data targeted by ODIN is Interlinear Glossed Text, or IGT, a semi-structured data format for presentation of linguistic examples, as shown in Figure 1. This data has been shown to have interest- ing characteristics, making some NLP analysis possi- ble for resource-poor languages (Lewis and Xia, 2008; Georgi et al., 2014; Georgi et al., 2015). Figure 1: An IGT instance of Kurmanji (kmr) in situ from (van de Visser, 2006). All the IGT instances in ODIN, including the one in Figure 1, were extracted from linguistic articles that were distributed electronically as Portable Docu- ment Format (PDF) documents, a format developed by Initial Page First Cut Second Cut Third Cut Figure 2: An illustration of the XY-Cut algorithm re- cursing on a document. The regions created by the current cut (if any) are represented by a dotted border, while regions not under consideration at the current recursion are shaded. Adobe Systems for the purposes of standardizing doc- ument display and typesetting across platforms. As this format was primarily designed to “communicate visual material between different computer applica- tions and systems” (Warnock, 1991), re-extracting the displayed text for automated systems was not an in- tended aspect of the format design. In the PDF spec- ification, text is represented as glyphs of a specified font in a “Character Space” coordinate system, em- bedded within a “Text Space” renderer (Bienz et al., 1997). One of the downsides of this document struc- ture is that the internal structure of a PDF file gives text as glyph-coordinate pairs, and guarantees only the positioning of text rendered on the page. It does not guarantee that the order of the encoded text resembles the order in which it is intended to be read. Consequently, extracting text …

Content truncated to 500 words