VILA: Improving Structured Content Extraction from Scientific PDFs Using Visual Layout Groups

VILA: Improving Structured Content Extraction from Scientific PDFs Using Visual Layout Groups Zejiang Shen1 Kyle Lo1 Lucy Lu Wang1 Bailey Kuehl1 Daniel S. Weld1,2 Doug Downey1,3 1Allen Institute for AI, USA 2University of Washington, USA 3Northwestern University, USA {shannons,kylel,lucyw,baileyk,danw,dougd}@allenai.org Abstract Accurately extracting structured content from PDFs is a critical first step for NLP over scientific papers. Recent work has improved extraction accuracy by incorporating elemen- tary layout information, for example, each token’s 2D position on the page, into language model pretraining. We introduce new methods that explicitly model VIsual LAyout (VILA) groups, that is, text lines or text blocks, to further improve performance. In our I-VILA approach, we show that simply inserting spe- cial tokens denoting layout group boundaries into model inputs can lead to a 1.9% Macro F1 improvement in token classification. In the H-VILA approach, we show that hierarchical encoding of layout-groups can result in up to 47% inference time reduction with less than 0.8% Macro F1 loss. Unlike prior layout-aware approaches, our methods do not require expen- sive additional pretraining, only fine-tuning, which we show can reduce training cost by up to 95%. Experiments are conducted on a newly curated evaluation suite, S2-VLUE, that unifies existing automatically labeled datasets and includes a new dataset of man- ual annotations covering diverse papers from 19 scientific disciplines. Pre-trained weights, benchmark datasets, and source code are avail- able at https://github.com/allenai /VILA. 1 Introduction Scientific papersareusuallydistributedinPortable Document Format (PDF) without extensive se- mantic markup. Extracting structured document representations from these PDF files—i.e., identi- fying title and author blocks, figures, references, and so on—is a critical first step for downstream NLP tasks (Beltagy et al., 2019; Wang et al., 2020) and is important for improving PDF accessibility (Wang et al., 2021). Recent work demonstrates that document lay- out information can be used to enhance content extraction via large-scale, layout-aware pre- training (Xu et al., 2020, 2021; Li et al., 2021). However, these methods only consider individual tokens’ 2D positions and do not ex- plicitly model high-level layout structures like the grouping of text into lines and blocks (see Figure 1 for an example), limiting accuracy. Further, existing methods come with enormous computational costs: They rely on further pre- training an existing pretrained model like BERT (Devlin et al., 2019) on layout-enriched in- put, and achieving the best performance from the models requires more than a thousand (Xu et al., 2020) to several thousand (Xu et al., 2021) GPU-hours. This means that swapping in a new pretrained text model or experimenting with new layout-aware architectures can be prohibitively expensive, incompatible with the goals of green AI (Schwartz et al., 2020). In this paper, we explore how to improve the accuracy and efficiency of structured con- tent extraction from scientific documents by using VIsual LAyout (VILA) groups. Following Zhong et al. (2019) and Tkaczyk et al. (2015), our methods use the idea that a document page can be segmented into visual groups of tokens (either lines or blocks), and that the tokens within each …

Content truncated to 500 words