GMN: Generative Multi-modal Network for Practical Document Information Extraction

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3768 - 3778 July 10-15, 2022 ©2022 Association for Computational Linguistics GMN: Generative Multi-modal Network for Practical Document Information Extraction Haoyu Cao1, Jiefeng Ma2∗, Antai Guo1, Yiqing Hu1, Hao Liu1, Deqiang Jiang1, Yinsong Liu1, Bo Ren1 1Tencent YouTu Lab 2University of Science and Technology of China {rechycao,ankerguo,hooverhu,ivanhliu,dqiangjiang}@tencent.com jfma@mail.ustc.edu.cn {jasonysliu,timren}@tencent.com Abstract Document Information Extraction (DIE) has attracted increasing attention due to its various advanced applications in the real world. Al- though recent literature has already achieved competitive results, these approaches usually fail when dealing with complex documents with noisy OCR results or mutative layouts. This paper proposes Generative Multi-modal Network (GMN) for real-world scenarios to ad- dress these problems, which is a robust multi- modal generation method without predefined label categories. With the carefully designed spatial encoder and modal-aware mask mod- ule, GMN can deal with complex documents that are hard to serialized into sequential or- der. Moreover, GMN tolerates errors in OCR results and requires no character-level annota- tion, which is vital because fine-grained anno- tation of numerous documents is laborious and even requires annotators with specialized do- main knowledge. Extensive experiments show that GMN achieves new state-of-the-art per- formance on several public DIE datasets and surpasses other methods by a large margin, es- pecially in realistic scenes. 1 Introduction Document Information Extraction (DIE) aims to map each document to a structured form consistent with the target ontology (e.g., database schema), which has recently become an increasingly impor- tant task. Recent research (Xu et al., 2020, 2021; Wang et al., 2021a; Zhang et al., 2020; Li et al., 2021a) has achieved competitive results for infor- mation extraction in the idealized scenario with accurate OCR results, word-level annotations, and serialized document words in reading order. These methods regard DIE as a Sequence Labeling (SL) task. Given OCR results of a document image, the traditional sequence labeling method first serializes ∗Work is done during an internship at Tencent YouTu Lab. Sequential order Reading order Name : TAHU DOMBA MEMBARA 1 TAHU DOMBA 13,636 MEMBARA B-Num B-Name I-Name B-Price I-Name Name - 1 Name - 2 Name: TAHU DOMBA | Name: MEMBARA SL GMN Fig1 Total: 24,000 | CASH: 24,000 Duplicated value TOTAL 24,000 CASH 24,000 O Total - 1 B-Total O B-Total Total - 2 Total : 24,000 | Total:24,000 SL GMN Fig4 Confused key Subtotal: 15.90 | Total Excl. of GST: 15.90 Subtotal : 15.90 | Total:15.90 Subtotal 15.90 Total Excl. of GST 15.00 O B-Subtotal O O O O B-Total SL GMN Fig3 Total - 2 Total - 2 Company : GUCCI G UC C l Bicaster B-Com Company - 1 I-Com I-Com I-Com O Company : GUUC1 SL GMN Fig2 Nosiy OCR GUCCI GUCC1 Figure 1: Examples in public DIE benchmarks with practical problems. The three rows from top to bottom in each sub-figure are 1) input images, 2) raw intermediate tags and final results generated by the SL …

Content truncated to 500 words