Combining OCR Outputs for Logical Document Structure Markup. Technical Background to the ACL 2012 Contributed Task

Proceedings of the ACL-2012 Special Workshop on Rediscovering 50 Years of Discoveries, pages 104–109, Jeju, Republic of Korea, 10 July 2012. c⃝2012 Association for Computational Linguistics Combining OCR Outputs for Logical Document Structure Markup Technical Background to the ACL 2012 Contributed Task Ulrich Schäfer Benjamin Weitz DFKI Language Technology Lab Campus D 3 1, D-66123 Saarbrücken, Germany {ulrich.schaefer|benjamin.weitz}@dfki.de Abstract We describe how paperXML, a logical docu- ment structure markup for scholarly articles, is generated on the basis of OCR tool out- puts. PaperXML has been initially developed for the ACL Anthology Searchbench. The main purpose was to robustly provide uni- form access to sentences in ACL Anthology papers from the past 46 years, ranging from scanned, typewriter-written conference and workshop proceedings papers, up to recent high-quality typeset, born-digital journal arti- cles, with varying layouts. PaperXML markup includes information on page and paragraph breaks, section headings, footnotes, tables, captions, boldface and italics character styles as well as bibliographic and publication meta- data. The role of paperXML in the ACL Con- tributed Task Rediscovering 50 Years of Dis- coveries is to serve as fall-back source (1) for older, scanned papers (mostly published be- fore the year 2000), for which born-digital PDF sources are not available, (2) for born- digital PDF papers on which the PDFExtract method failed, (3) for document parts where PDFExtract does not output useful markup such as currently for tables. We sketch trans- formation of paperXML into the ACL Con- tributed Task’s TEI P5 XML. 1 Introduction Work on the ACL Anthology Searchbench started in 2009. The goal was to provide combined sentence- semantic, full-text and bibliographic search in the complete ACL Anthology (Schäfer et al., 2011), and a graphical citation browser with citation sentence context information (Weitz & Schäfer, 2012). Since the ACL-HLT 2011 conference, the Searchbench is available as a free, public service1. A fixed subset of the Anthology, the ACL An- thology Reference Corpus2 (ACL-ARC), contains various representations of the papers such as PDF, bitmap and text files. The latter were generated with PDFBox3 and OCR (Omnipage4), applied to the PDF files or bitmap versions thereof. Its static nature as infrequently released reference corpus and low character recognition quality especially of older, badly scanned papers, made us to look for alterna- tives. For quick, automatic updates of the Search- bench index, a robust method for getting the text from old and new incoming PDF files was needed. After a thorough comparison of different PDF-to- text extraction tools, a decision was made to process every PDF paper in the Anthology with ABBYY PDF Transformer5, for various reasons. It ran stably and delivered good character recognition rates on both scanned, typewriter-typeset proceeding papers as well as on born-digital PDF of various sources, even on papers where PDFbox failed to extract (us- able) text. Reading order recovery, table recognition and output rendering (HTML) was impressive and de-hyphenation for English text worked reasonably well. All in all, ABBYY did not deliver perfect re- sults, but at that time was the …

Content truncated to 500 words