Entity Relation Extraction as Dependency Parsing in Visually Rich Documents

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 2759–2768 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 2759 Entity Relation Extraction as Dependency Parsing in Visually Rich Documents Yue Zhang1,Bo Zhang1, Rui Wang2∗, Junjie Cao1, Chen Li1, Zuyi Bao1 Alibaba Group, China 1{shiyu.zy,klayzhang.zb,puji.lc,junjie.junjiecao,zuyi.bzy} @alibaba-inc.com 2mars198356@hotmail.com Abstract Previous works on key information extraction from visually rich documents (VRDs) mainly focus on labeling the text within each bound- ing box (i.e., semantic entity), while the rela- tions in-between are largely unexplored. In this paper, we adapt the popular dependency parsing model, the biaffine parser, to this entity relation extraction task. Being dif- ferent from the original dependency parsing model which recognizes dependency relations between words, we identify relations between groups of words with layout information in- stead. We have compared different representa- tions of the semantic entity, different VRD en- coders, and different relation decoders. For the model training, we explore multi-task learning to combine entity labeling and relation extrac- tion tasks; and for the evaluation, we conduct experiments on different datasets with filtering and augmentation. The results demonstrate that our proposed model achieves 65.96% F1 score on the FUNSD dataset. As for the real- world application, our model has been applied to the in-house customs data, achieving reli- able performance in the production setting. 1 Introduction In real-life scenarios, there are many types of vi- sually rich documents (VRDs), such as invoices, questionnaire forms, declaration materials and so on. These documents contain abundant layout in- formation which helps us to understand the content while texts alone are not enough. In recent years, many works focus on how to extract key infor- mation from VRDs based on the results of OCR (Optical Character Recognition), which recognizes bounding boxes and texts within the boxes (Liu et al., 2019a; Yu et al., 2020b). Each bounding box contains 1) a group of words that belong together ∗Corresponding author. The author’s contributions were carried out while at Alibaba Group. His current affiliation is Vipshop (China) Co., Ltd. from a semantic and spatial standpoint and 2) vi- sual features such as layout, tabular structure and font size of the boxes in the document. We call such bounding boxes and texts within the boxes se- mantic entities1, and each entity contains the word group and layout coordinates2. Key information extraction (KIE) is such a task to analyze visually rich documents, which usually contains two steps, entity labeling and entity rela- tion extraction. Similar to named entity recognition (NER) and relation extraction in the traditional nat- ural language processing (NLP), entity labeling aims to assign predefined labels to the semantic entities in VRDs (G. Jaume and Thiran, 2019; Liu et al., 2019a; Yu et al., 2020b), and entity relation extraction3 predicts relations between these seman- tic entities. Compared with NER and relation ex- traction in the traditional NLP, KIE from VRDs is a more challenging task. First, a normal (named-) entity in plain text does not contain layout informa- tion as those semantic entities …

Content truncated to 500 words