Read Extensively, Focus Smartly: A Cross-document Semantic Enhancement Method for Visual Documents NER

Proceedings of the 29th International Conference on Computational Linguistics, pages 2034–2043 October 12–17, 2022. 2034 Read Extensively, Focus Smartly: A Cross-document Semantic Enhancement Method for Visual Documents NER Jun Zhao1∗, Xin Zhao1∗, Wenyu Zhan1, Tao Gui2†, Qi Zhang1†, Liang Qiao3, Zhanzhan Cheng3, Shiliang Pu3 1School of Computer Science, Shanghai Key Laboratory of Intelligent Information Processing, Fudan University, Shanghai, China 2Institute of Modern Languages and Linguistics, Fudan University 3Hikvision Research Institute, Shanghai, China {zhaoj19,tgui,qz}@fudan.edu.cn,{zhaoxin21,wyzhan21}@m.fudan.edu.cn Abstract The introduction of multimodal information and pretraining technique significantly im- proves entity recognition from visually-rich documents. However, most of the existing methods pay unnecessary attention to irrele- vant regions of the current document while ig- noring the potentially valuable information in related documents. To deal with this problem, this work proposes a cross-document semantic enhancement method, which consists of two modules: 1) To prevent distractions from irrelevant regions in the current document, we design a learnable attention mask mechanism, which is used to adaptively filter redundant information in the current document. 2) To further enrich the entity-related context, we propose a cross-document information aware- ness technique, which enables the model to collect more evidence across documents to assist in prediction. The experimental results on two documents understanding benchmarks covering eight languages demonstrate that our method outperforms the SOTA methods. 1 Introduction Visually-rich documents (VRDs) are the most common information carriers in the real world, such as newspapers, resumes, tickets, etc. Differ- ent from plain text data, the information in VRDs is encoded by multiple modalities including text, vision, and layout. Entity recognition from VRDs, as a key step for document understanding, is of utmost practical interest for many downstream applications such as business analysis (Xu et al., 2020), intelligent education (Kahraman et al., 2010), digital library (Kroll et al., 2021). The pioneering explorations approach this task by either computer vision (Katti et al., 2018) or natural language processing (Lample et al., 2016) *Equal Contributions. †Corresponding authors. The New York Times. COVID-19 spreads in the US. Facebook renames itself meta. Target-Related Context Non-Related Context What is COVID-19? COVID-19 spreads in the US. Fever is a sympton of COVID-19. Omicron is a variant of COVID-19. COVID-19 causes pneumonia. This is the name of a virus! Figure 1: Redundant irrelevant information and insuf- ficient entity-related information in current document make it difficult to extract entities accurately. We filter redundant information from irrelevant regions and expand the information source from a single to multiple related documents. Thereby, entity-related information is efficiently integrated and the extraction is improved. paradigm. However, these methods ignore the in- herent multi-modality of VRDs and consequently the suboptimal results are achieved. To address this problem, graph neural network (Liu et al., 2019) and self-attention mechanism (Zhang et al., 2020) are introduced to capture the cross-modality interaction and achieve superior performance. Re- cently, inspired by the widespread success of large pretrained language models (Qiu et al., 2020), self-supervised pretraining techniques are leveraged to learn cross-modal knowledge from unlabeled documents (Xu et al., 2020, 2021a,b) and the amount …

Content truncated to 500 words