MarkupLM: Pre-training of Text and Markup Language for Visually Rich Document Understanding

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 6078 - 6087 May 22-27, 2022 c⃝2022 Association for Computational Linguistics MarkupLM: Pre-training of Text and Markup Language for Visually Rich Document Understanding Junlong Li1∗, Yiheng Xu2∗, Lei Cui2, Furu Wei2 1Shanghai Jiao Tong University 2Microsoft Research Asia lockonn@sjtu.edu.cn {t-yihengxu,lecu,fuwei}@microsoft.com Abstract Multimodal pre-training with text, layout, and image has made significant progress for Visu- ally Rich Document Understanding (VRDU), especially the fixed-layout documents such as scanned document images. While, there are still a large number of digital documents where the layout information is not fixed and needs to be interactively and dynamically rendered for visualization, making existing layout-based pre-training approaches not easy to apply. In this paper, we propose MarkupLM for doc- ument understanding tasks with markup lan- guages as the backbone, such as HTML/XML- based documents, where text and markup in- formation is jointly pre-trained. Experiment results show that the pre-trained MarkupLM significantly outperforms the existing strong baseline models on several document under- standing tasks. The pre-trained model and code will be publicly available at https:// aka.ms/markuplm. 1 Introduction Multimodal pre-training with text, layout, and vi- sual information has recently become the de facto approach (Xu et al., 2020, 2021a,b; Pramanik et al., 2020; Garncarek et al., 2021; Hong et al., 2021; Powalski et al., 2021; Wu et al., 2021; Li et al., 2021a,b; Appalaraju et al., 2021) in Visually-rich Document Understanding (VRDU) tasks. These multimodal models are usually pre-trained with the Transformer architecture (Vaswani et al., 2017) us- ing large-scale unlabeled scanned document im- ages (Lewis et al., 2006) or digital-born PDF files, followed by task-specific fine-tuning with relatively small-scale labeled training samples to achieve the state-of-the-art performance on a va- riety of document understanding tasks, including form understanding (Jaume et al., 2019; Xu et al., 2021b), receipt understanding (Huang et al., 2019; ∗Equal contributions during internship at Microsoft Re- search Asia. Corresponding authors: Lei Cui and Furu Wei Park et al., 2019), complex document understand- ing (Grali´nski et al., 2020), document type classifi- cation (Harley et al., 2015), and document visual question answering (Mathew et al., 2021), etc. Sig- nificant progress has been witnessed not only in research tasks within academia, but also in different real-world business applications such as finance, insurance, and many others. Visually rich documents can be generally di- vided into two categories. The first one is the fixed-layout documents such as scanned document images and digital-born PDF files, where the lay- out and style information is pre-rendered and in- dependent of software, hardware, or operating sys- tem. This property makes existing layout-based pre-training approaches easily applicable to docu- ment understanding tasks. While, the second cat- egory is the markup-language-based documents such as HTML/XML, where the layout and style information needs to be interactively and dynami- cally rendered for visualization depending on the software, hardware, or operating system, which is shown in Figure 1. For markup-language-based documents, the 2D layout information does not ex- ist in an …

Content truncated to 500 words