LayoutMask: Enhance Text-Layout Interaction in Multi-modal Pre-training for Document Understanding

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 15200–15212 July 9-14, 2023 ©2023 Association for Computational Linguistics LayoutMask: Enhance Text-Layout Interaction in Multi-modal Pre-training for Document Understanding Yi Tu, Ya Guo, Huan Chen, Jinyang Tang Ant Group, China {qianyi.ty,guoya.gy,chenhuan.chen,jinyang.tjy}@antgroup.com Abstract Visually-rich Document Understanding (VrDU) has attracted much research attention over the past years. Pre-trained models on a large number of document images with transformer-based backbones have led to significant performance gains in this field. The major challenge is how to fusion the different modalities (text, layout, and image) of the documents in a unified model with different pre-training tasks. This paper focuses on improving text-layout interactions and proposes a novel multi-modal pre-training model, LayoutMask. LayoutMask uses local 1D position, instead of global 1D position, as layout input and has two pre-training objectives: (1) Masked Language Modeling: predicting masked tokens with two novel masking strategies; (2) Masked Position Modeling: predicting masked 2D positions to improve layout representation learning. LayoutMask can enhance the interactions between text and layout modalities in a unified model and produce adaptive and robust multi- modal representations for downstream tasks. Experimental results show that our proposed method can achieve state-of-the-art results on a wide variety of VrDU problems, including form understanding, receipt understanding, and document image classification. 1 Introduction Visually-rich Document Understanding (VrDU) is an important research area that aims to understand various types of documents (e.g., forms, receipts, and posters), and it has attracted much attention from both academia and industry. In recent years, pre-training techniques (Devlin et al., 2019; Zhang et al., 2019) have been introduced into this area and self-supervised pre-training multi-modal mod- els have demonstrated great successes in various VrDU tasks (Xu et al., 2020, 2021; Hong et al., 2022; Li et al., 2021a). Figure 1: A receipt image from SROIE dataset and the global/local 1D positions of tokens based on global/in- segment reading orders. Local 1D positions restart with “1” for each individual segment. Blue Arrow: When using global 1D position, the reading order is explicitly implied by the ascending numbers, so the word after “Qty” is “Price”. Red Arrows: When using local 1D position, the successor of “Qty” is not directly given and can have more possible choices, so their semantic relations and 2D positions will be considered during pre-training. However, existing document pre-training mod- els suffer from reading order issues. Following the idea of BERT (Devlin et al., 2019), these methods (Xu et al., 2020, 2021; Hong et al., 2022) usually adopt ascending numbers (e.g., 0, 1, 2,.., 511) to represent the global reading order of tokens in the document. Then, these numbers are encoded into 1D position embeddings to provide explicit reading order supervision during pre-training, which are called “global 1D position”. While such global 1D positions are widely used in NLP models for textual data, it is not a good choice for document data. Firstly, plain texts always have definite and linear reading orders, but the reading order of a …

Content truncated to 500 words