FormNetV2: Multimodal Graph Contrastive Learning for Form Document Information Extraction

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 9011–9026 July 9-14, 2023 ©2023 Association for Computational Linguistics FormNetV2: Multimodal Graph Contrastive Learning for Form Document Information Extraction Chen-Yu Lee1∗, Chun-Liang Li1, Hao Zhang2, Timothy Dozat2, Vincent Perot2, Guolong Su2, Xiang Zhang1, Kihyuk Sohn2, Nikolai Glushnev3, Renshen Wang2, Joshua Ainslie2, Shangbang Long2, Siyang Qin2, Yasuhisa Fujii2, Nan Hua2, Tomas Pfister1 1Google Cloud AI Research, 2Google Research, 3Google Cloud AI Abstract The recent advent of self-supervised pre- training techniques has led to a surge in the use of multimodal learning in form docu- ment understanding. However, existing ap- proaches that extend the mask language mod- eling to other modalities require careful multi- task tuning, complex reconstruction target de- signs, or additional pre-training data. In Form- NetV2, we introduce a centralized multimodal graph contrastive learning strategy to unify self-supervised pre-training for all modalities in one loss. The graph contrastive objective maximizes the agreement of multimodal repre- sentations, providing a natural interplay for all modalities without special customization. In addition, we extract image features within the bounding box that joins a pair of tokens con- nected by a graph edge, capturing more tar- geted visual cues without loading a sophisti- cated and separately pre-trained image embed- der. FormNetV2 establishes new state-of-the- art performance on FUNSD, CORD, SROIE and Payment benchmarks with a more com- pact model size. 1 Introduction Automated information extraction is essential for many practical applications, with form-like doc- uments posing unique challenges compared to article-like documents, which have led to an abun- dance of recent research in the area. In particular, form-like documents often have complex layouts that contain structured objects like tables, columns, and fillable regions. Layout-aware language model- ing has been critical for many successes (Xu et al., 2020; Majumder et al., 2020; Lee et al., 2022). To further boost the performance, many recent approaches adopt multiple modalities (Xu et al., ∗All work done at Google. Correspondence to: Chen- Yu Lee <chenyulee@google.com>, Chun-Liang Li <chun- liang@google.com> 2021; Huang et al., 2022; Appalaraju et al., 2021). Specifically, the image modality adds more struc- tural information and visual cues to the existing layout and text modalities. They therefore extend the masked language modeling (MLM) from text to masked image modeling (MIM) for image and text- image alignment (TIA) for cross-modal learning. The alignment objective may also help to prime the layout modality, though it does not directly involve text layouts or document structures. In this work, we propose FormNetV2, a mul- timodal transformer model for form information extraction. Unlike existing works – which may use the whole image as one representation (Appalaraju et al., 2021), or image patches (Xu et al., 2021), or image features of token bounding boxes (Xu et al., 2020) – we propose using image features extracted from the region bounded by a pair of tokens con- nected in the constructed graph. This allows us to capture a richer and more targeted visual compo- nent of the intra- and …

Content truncated to 500 words