StructuralLM: Structural Pre-training for Form Understanding

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 6309–6318 August 1–6, 2021. ©2021 Association for Computational Linguistics 6309 StructuralLM: Structural Pre-training for Form Understanding Chenliang Li, Bin Bi, Ming Yan, Wei Wang, Songfang Huang, Fei Huang, Luo Si Alibaba Group {lcl193798, b.bi, ym119608, hebian.ww}@alibaba-inc.com {songfang.hsf, f.huang, luo.si}@alibaba-inc.com Abstract Large pre-trained language models achieve state-of-the-art results when fine-tuned on downstream NLP tasks. However, they almost exclusively focus on text-only representation, while neglecting cell-level layout information that is important for form image understanding. In this paper, we propose a new pre-training ap- proach, StructuralLM, to jointly leverage cell and layout information from scanned docu- ments. Specifically, we pre-train StructuralLM with two new designs to make the most of the interactions of cell and layout information: 1) each cell as a semantic unit; 2) classification of cell positions. The pre-trained StructuralLM achieves new state-of-the-art results in differ- ent types of downstream tasks, including form understanding (from 78.95 to 85.14), docu- ment visual question answering (from 72.59 to 83.94) and document image classification (from 94.43 to 96.08). 1 Introduction Document understanding is an essential problem in NLP, which aims to read and analyze textual documents. In addition to plain text, many real- world applications require to understand scanned documents with rich text. As shown in Figure 1, such scanned documents contain various structured information, like tables, digital forms, receipts, and invoices. The information of a document image is usually presented in natural language, but the format can be organized in many ways from multi- column layout to various tables/forms. Inspired by the recent development of pre- trained language models (Devlin et al., 2019; Liu et al., 2019; Wang et al., 2019) in various NLP tasks, recent studies on document image pre- training (Zhang et al., 2020; Xu et al., 2019) have pushed the limits of a variety of document image understanding tasks, which learn the interaction be- tween text and layout information across scanned document images. Xu et al. (2019) propose LayoutLM, which is a pre-training method of text and layout for doc- ument image understanding tasks. It uses 2D- position embeddings to model the word-level lay- out information. However, it is not enough to model the word-level layout information, and the model should consider the cell as a semantic unit. It is important to know which words are from the same cell and to model the cell-level layout in- formation. For example, as shown in Figure 1 (a), which is from form understanding task (Jaume et al., 2019), determining that the ”LORILLARD” and the ”ENTITIES” are from the same cell is critical for semantic entity labeling. The ”LORIL- LARD ENTITIES” should be predicted as Answer entity, but LayoutLM predicts ”LORILLARD” and ”ENTITIES” as two separate entities. The input to traditional natural language tasks is usually presented as plain text, and text-only mod- els need to obtain the semantic representation of the input sentences and the semantic relationship between …

Content truncated to 500 words