LayoutLMv2: Multi-modal Pre-training for Visually-rich Document Understanding

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 2579–2591 August 1–6, 2021. ©2021 Association for Computational Linguistics 2579 LayoutLMv2: Multi-modal Pre-training for Visually-rich Document Understanding Yang Xu1∗, Yiheng Xu2 ∗, Tengchao Lv2 ∗, Lei Cui2, Furu Wei2, Guoxin Wang3, Yijuan Lu3, Dinei Florencio3, Cha Zhang3, Wanxiang Che1, Min Zhang4, Lidong Zhou2 1Research Center for Social Computing and Information Retrieval, Harbin Institute of Technology 2Microsoft Research Asia 3Microsoft Azure AI 1{yxu,car}@ir.hit.edu.cn,4Soochow University 2{v-yixu,v-telv,lecu,fuwei,lidongz}@microsoft.com, 3{guow,yijlu,dinei,chazhang}@microsoft.com 4minzhang@suda.edu.cn Abstract Pre-training of text and layout has proved effective in a variety of visually-rich docu- ment understanding tasks due to its effec- tive model architecture and the advantage of large-scale unlabeled scanned/digital-born documents. We propose LayoutLMv2 archi- tecture with new pre-training tasks to model the interaction among text, layout, and image in a single multi-modal framework. Specif- ically, with a two-stream multi-modal Trans- former encoder, LayoutLMv2 uses not only the existing masked visual-language model- ing task but also the new text-image align- ment and text-image matching tasks, which make it better capture the cross-modality in- teraction in the pre-training stage. Meanwhile, it also integrates a spatial-aware self-attention mechanism into the Transformer architecture so that the model can fully understand the relative positional relationship among differ- ent text blocks. Experiment results show that LayoutLMv2 outperforms LayoutLM by a large margin and achieves new state-of- the-art results on a wide variety of down- stream visually-rich document understanding tasks, including FUNSD (0.7895 → 0.8420), CORD (0.9493 → 0.9601), SROIE (0.9524 → 0.9781), Kleister-NDA (0.8340 → 0.8520), RVL-CDIP (0.9443 → 0.9564), and DocVQA (0.7295 → 0.8672). We made our model and code publicly available at https://aka.ms /layoutlmv2. 1 Introduction Visually-rich Document Understanding (VrDU) aims to analyze scanned/digital-born business doc- uments (images of invoices, forms in PDF format, etc.) where structured information can be automat- ically extracted and organized for many business ∗Equal contributions during internship at MSRA applications. Distinct from conventional informa- tion extraction tasks, the VrDU task relies on not only textual information but also visual and lay- out information that is vital for visually-rich docu- ments. Different types of documents indicate that the text fields of interest located at different posi- tions within the document, which is often deter- mined by the style and format of each type as well as the document content. Therefore, to accurately recognize the text fields of interest, it is inevitable to take advantage of the cross-modality nature of visually-rich documents, where the textual, visual, and layout information should be jointly modeled and learned end-to-end in a single framework. The recent progress of VrDU lies primarily in two directions. The first direction is usually built on the shallow fusion between textual and visual/layout/style information (Yang et al., 2017; Liu et al., 2019; Sarkhel and Nandi, 2019; Yu et al., 2020; Majumder et al., 2020; Wei et al., 2020; Zhang et al., 2020). These approaches leverage the pre-trained NLP and CV models individually and combine the information from multiple …

Content truncated to 500 words