Wukong-Reader: Multi-modal Pre-training for Fine-grained Visual Document Understanding

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 13386–13401 July 9-14, 2023 ©2023 Association for Computational Linguistics WUKONG-READER: Multi-modal Pre-training for Fine-grained Visual Document Understanding Haoli Bai∗, Zhiguang Liu∗, Xiaojun Meng∗, Wentao Li, Shuang Liu, Yifeng Luo, Nian Xie, Rongfu Zheng, Liangwei Wang†, Lu Hou†, Jiansheng Wei, Xin Jiang, Qun Liu Noah’s Ark Lab, Huawei Technologies {baihaoli, liuzhiguang1, xiaojun.meng, liwentao18, liushuang30, luoyifeng1, xienian, zhengrongfu, wangliangwei, houlu3, weijiansheng, Jiang.Xin, qun.liu}@huawei.com Abstract Unsupervised pre-training on millions of digital-born or scanned documents has shown promising advances in visual document un- derstanding (VDU). While various vision- language pre-training objectives are studied in existing solutions, the document textline, as an intrinsic granularity in VDU, has seldom been explored so far. A document textline usu- ally contains words that are spatially and se- mantically correlated, which can be easily ob- tained from OCR engines. In this paper, we propose WUKONG-READER, trained with new pre-training objectives to leverage the structural knowledge nested in document textlines. We introduce textline-region contrastive learning to achieve fine-grained alignment between the vi- sual regions and texts of document textlines. Furthermore, masked region modeling and textline-grid matching are also designed to en- hance the visual and layout representations of textlines. Experiments show that WUKONG- READER brings superior performance on var- ious VDU tasks in both English and Chinese. The fine-grained alignment over textlines also empowers WUKONG-READER with promising localization ability. 1 Introduction Visual document understanding (VDU) handles var- ious types of digital-born or scanned documents like forms, tables, reports, or research papers, and is becoming increasingly important for real-world industrial practices [7]. Multi-modal pre-training on millions of documents is a popular solution for visual document understanding [12, 33, 35, 34, 14, 27]. Unlike the conventional vision-language pre-training over natural images and their paired short and abstractive descriptions [29, 22, 21], the document texts are usually long and highly cor- related with the images, since they can be easily ∗Equal Contribution. †Corresponding authors. (a) Letter. (b) Receipt. Figure 1: Samples of document textlines: a) a letter from FUNSD[16] with keys (blue) and values (green); and b) a receipt from SROIE [15] with the restaurants name (dark blue) and the address (orange). obtained from accurate Optical Character Recog- nition (OCR) engines from the scanned images. Therefore, it is crucial to strengthen the connection between vision and language for VDU with more fine-grained alignment across the two modalities. Towards that end, existing efforts seek to align the visual and textual knowledge of documents at different levels. A commonly used pre-training objective for documents is masked language mod- eling [8] over document text tokens [35, 34, 14, 33, 12, 27], often accompanied by the layout in- formation encoded via the positional embedding. Besides, various visual and vision-language mul- timodal pre-training objectives are also proposed, leveraging the patch-level features [34, 14], object- level features from object detectors [23, 11], or the whole image feature through a global text-image matching loss [34]. However, as an intrinsic granularity for VDU, document …

Content truncated to 500 words