MGDoc: Pre-training with Multi-granular Hierarchy for Document Image Understanding

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 3984 - 3993 December 7-11, 2022 ©2022 Association for Computational Linguistics MGDoc: Pre-training with Multi-granular Hierarchy for Document Image Understanding Zilong Wang1∗ Jiuxiang Gu2 Chris Tensmeyer2 Nikolaos Barmpalios2 Ani Nenkova2 Tong Sun2 Jingbo Shang1 Vlad I. Morariu2 1University of California, San Diego 2Adobe Research 1{zlwang, jshang}@ucsd.edu 2{jigu, tensmeye, barmpali, nenkova, tsun, morariu}@adobe.com Abstract Document images are a ubiquitous source of data where the text is organized in a complex hierarchical structure ranging from fine granu- larity (e.g., words), medium granularity (e.g., regions such as paragraphs or figures), to coarse granularity (e.g., the whole page). The spatial hierarchical relationships between content at different levels of granularity are crucial for document image understanding tasks. Existing methods learn features from either word-level or region-level but fail to consider both simul- taneously. Word-level models are restricted by the fact that they originate from pure-text language models, which only encode the word- level context. In contrast, region-level mod- els attempt to encode regions corresponding to paragraphs or text blocks into a single embed- ding, but they perform worse with additional word-level features. To deal with these issues, we propose MGDoc, a new multi-modal multi- granular pre-training framework that encodes page-level, region-level, and word-level infor- mation at the same time. MGDoc uses a uni- fied text-visual encoder to obtain multi-modal features across different granularities, which makes it possible to project the multi-granular features into the same hyperspace. To model the region-word correlation, we design a cross- granular attention mechanism and specific pre- training tasks for our model to reinforce the model of learning the hierarchy between re- gions and words. Experiments demonstrate that our proposed model can learn better fea- tures that perform well across granularities and lead to improvements in downstream tasks. 1 Introduction Document images are ubiquitous and are often used as a representation for forms, receipts, printed pa- pers, etc. Unlike plain text documents, document images express rich information via both textual ∗ This work was completed while the author was working as an intern at Adobe Research. (a) Page-level (b) Region-level (c) Word-level Figure 1: The multi-granular structure in document im- ages. The image is from the RVL-CDIP dataset. Impor- tant information is encoded at the page-level (e.g., the document type), region-level (constructs such as para- graphs, tables, etc), and word-level (specific semantics). Our proposed model reasons about all of these jointly to leverage information across granularities. content and heterogeneous layout patterns, which leads to barriers to the automatic processing of these document images. Here, layout pattern refers to how the text is spatially arranged on the docu- ment page and involves information from multiple levels of granularity. Specifically, a layout pattern divides the entire page into individual regions and, within each region, the fine-grained textual content is distributed following a certain format, such as paragraphs, columns, lists, as shown in Figure 1. The layout of a document provides important cues for interpreting the document through spatial structures such …

Content truncated to 500 words