FormNet: Structural Encoding beyond Sequential Modeling in Form Document Information Extraction

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 3735 - 3754 May 22-27, 2022 c⃝2022 Association for Computational Linguistics FormNet: Structural Encoding beyond Sequential Modeling in Form Document Information Extraction Chen-Yu Lee†, Chun-Liang Li†, Timothy Dozat‡, Vincent Perot‡, Guolong Su‡, Nan Hua‡, Joshua Ainslie‡, Renshen Wang‡, Yasuhisa Fujii‡, Tomas Pfister† †Google Cloud AI Research, ‡Google Research {chenyulee, chunliang, tdozat, vperot, gsu, nhua, jainslie, rewang, yasuhisaf, tpfister}@google.com Abstract Sequence modeling has demonstrated state-of- the-art performance on natural language and document understanding tasks. However, it is challenging to correctly serialize tokens in form-like documents in practice due to their variety of layout patterns. We propose Form- Net, a structure-aware sequence model to mit- igate the suboptimal serialization of forms. First, we design Rich Attention that leverages the spatial relationship between tokens in a form for more precise attention score calcu- lation. Second, we construct Super-Tokens for each word by embedding representations from their neighboring tokens through graph convolutions. FormNet therefore explicitly re- covers local syntactic information that may have been lost during serialization. In experi- ments, FormNet outperforms existing methods with a more compact model size and less pre- training data, establishing new state-of-the-art performance on CORD, FUNSD and Payment benchmarks. 1 Introduction Form-like document understanding is a surging re- search topic because of its practical applications in automating the process of extracting and organiz- ing valuable text data sources such as marketing documents, advertisements and receipts. Typical documents are represented using natu- ral languages; understanding articles or web con- tent (Antonacopoulos et al., 2009; Luong et al., 2012; Soto and Yoo, 2019) has been studied exten- sively. However, form-like documents often have more complex layouts that contain structured ob- jects, such as tables and columns. Therefore, form documents have unique challenges compared to natural language documents stemming from their structural characteristics, and have been largely under-explored. In this work, we study critical information ex- traction from form documents, which is the funda- ��� ����� ������������ �������������������������������� ��������� �������� ������������� ��������� ���������������� ������������������������������������������������������������� Figure 1: An illustration of the form document infor- mation extraction task. mental subtask of form document understanding. Following the success of sequence modeling in natural language understanding (NLU), a natural approach to tackle this problem is to first serialize the form documents and then apply state-of-the- art sequence models to them. For example, Palm et al. (2017) use Seq2seq (Sutskever et al., 2014) with RNN, and Hwang et al. (2019) use transform- ers (Vaswani et al., 2017). However, interwoven columns, tables, and text blocks make serialization difficult, substantially limiting the performance of a strict serialization approach. To model the structural information present in documents, Katti et al. (2018); Zhao et al. (2019); Denk and Reisswig (2019) treat the documents as 2D image inputs and directly apply convolutional networks on them to preserve the spatial context during learning and inference. However, the perfor- mance is limited by the resolution of the 2D input grids. Another approach is a two-step pipeline …

Content truncated to 500 words