XFUND: A Benchmark Dataset for Multilingual Visually Rich Form Understanding

Findings of the Association for Computational Linguistics: ACL 2022, pages 3214 - 3224 May 22-27, 2022 c⃝2022 Association for Computational Linguistics XFUND: A Benchmark Dataset for Multilingual Visually Rich Form Understanding Yiheng Xu1∗, Tengchao Lv1, Lei Cui1, Guoxin Wang2, Yijuan Lu2, Dinei Florencio2, Cha Zhang2, Furu Wei1 1Microsoft Research Asia 2Microsoft Azure AI {t-yihengxu,tengchaolv,lecu}@microsoft.com {guow,yijlu,dinei,chazhang,fuwei}@microsoft.com Abstract Multimodal pre-training with text, layout, and image has achieved SOTA performance for vi- sually rich document understanding tasks re- cently, which demonstrates the great potential for joint learning across different modalities. However, the existed research work has focused only on the English domain while neglecting the importance of multilingual generalization. In this paper, we introduce a human-annotated multilingual form understanding benchmark dataset named XFUND, which includes form understanding samples in 7 languages (Chi- nese, Japanese, Spanish, French, Italian, Ger- man, Portuguese). Meanwhile, we present Lay- outXLM, a multimodal pre-trained model for multilingual document understanding, which aims to bridge the language barriers for vi- sually rich document understanding. Exper- imental results show that the LayoutXLM model has significantly outperformed the ex- isting SOTA cross-lingual pre-trained mod- els on the XFUND dataset. The XFUND dataset and pre-trained LayoutXLM models have been publicly available at https:// aka.ms/layoutxlm. 1 Introduction Recently, multimodal pre-training for visually rich document understanding (VRDU) has achieved new SOTA performance on several public bench- marks (Xu et al., 2021, 2020), including form un- derstanding (Jaume et al., 2019), receipt under- standing (Park et al., 2019), complex layout un- derstanding (Stanisławek et al., 2021), document image classification (Harley et al., 2015) and docu- ment VQA task (Mathew et al., 2021), due to the advantage that text, layout and image information is jointly learned end-to-end in a single framework. However, since most evaluation benchmarks focus ∗Contribution during internship at Microsoft Research Asia. Correspondence to Lei Cui<lecu@microsoft.com> and Furu Wei<fuwei@microsoft.com> on English VRDs, it is hard to explore the per- formance of a document understanding system on VRDs in other languages. Simply translating these documents automatically with machine translation services might help, but it is often not satisfactory due to the poor translation quality on document images (Afli and Way, 2016). Therefore, it is vital to explore the multilingual generalization ability of multimodal pre-training for VRDU tasks. Multilingual pre-trained models such as mBERT (Devlin et al., 2019), XLM (Conneau and Lample, 2019), XLM-RoBERTa (Conneau et al., 2020), mBART (Liu et al., 2020), and the recent InfoXLM (Chi et al., 2021) and mT5 (Xue et al., 2021) have pushed many SOTA results on cross-lingual natural language understanding tasks by pre-training the Transformer models on different languages. These models have successfully bridged the language barriers in a number of cross-lingual transfer benchmarks such as XNLI (Conneau et al., 2018) and XTREME (Hu et al., 2020). Although a large amount of multilingual text data has been used in these cross-lingual pre-trained models, text-only multilingual models cannot be easily used in the VRDU tasks because they are usually fragile in analyzing the documents due to the format/layout diversity of documents in …

Content truncated to 500 words