DocStruct: A Multimodal Method to Extract Hierarchy Structure in Document for General Form Understanding

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 898–908 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 898 DocStruct: A Multimodal Method to Extract Hierarchy Structure in Document for General Form Understanding Zilong Wang Mingjie Zhan Xuebo Liu Ding Liang Sensetime {wangzilong, zhanmingjie, liuxuebo, liangding}@sensetime.com Abstract Form understanding depends on both textual contents and organizational structure. Al- though modern OCR performs well, it is still challenging to realize general form under- standing because forms are commonly used and of various formats. The table detection and handcrafted features in previous works cannot apply to all forms because of their re- quirements on formats. Therefore, we concen- trate on the most elementary components, the key-value pairs, and adopt multimodal meth- ods to extract features. We consider the form structure as a tree-like or graph-like hierar- chy of text fragments. The parent-child re- lation corresponds to the key-value pairs in forms. We utilize the state-of-the-art models and design targeted extraction modules to ex- tract multimodal features from semantic con- tents, layout information, and visual images. A hybrid fusion method of concatenation and feature shifting is designed to fuse the hetero- geneous features and provide an informative joint representation. We adopt an asymmetric algorithm and negative sampling in our model as well. We validate our method on two bench- marks, MedForm and FUNSD, and extensive experiments demonstrate the effectiveness of our method. 1 Introduction Forms are a ubiquitous document format. Numer- ous forms are used in finance, insurance and medi- cal industry every day. Although forms vary a lot, we consider it as a collection of key-value pairs and all these pairs establish a hierarchical structure within the page. Our work in this paper focuses on utilizing multimodal information to extract the hier- archy from the forms. Equipped with the hierarchy, it is oversimplified to further analyze the general forms and extract the structural data. Modern Optical Character Recognition (OCR) has already provided a reliable and efficient way DIVISION: DIVISION NAME: Detroit North DIVISION NAME: Detroit East DIVISION: DIVISION NAME: DIVISION NAME: Detroit North Detroit East (a) (b) Check items Result Heart rate 72 Heart rhythm Regular [Medical examination] (c) (d) [Medical examination] Check items Result Heart rhythm Regular Heart rate 72 Figure 1: Example from FUNSD Dataset: (a) is part of the original image; (b) is the hierarchical structure. Ex- ample from MedForm Dataset: (c) is part of the origi- nal image; (d) is the hierarchical structure. for the computers to read the textual contents of form pages. The contents can be divided into sev- eral individual textual fragments. However, it is not enough to form-understanding tasks. The informa- tion is expressed not only through the textual data in each section, but also through the way in which the sections are organized. Some of the fragments serve as headers, topics or questions of their coun- terparts. We consider the relation as key-value pairs in a hierarchy. Figure 1 are two examples 1. There- fore, after preliminary processing with OCR, …

Content truncated to 500 words