LiLT: A Simple yet Effective Language-Independent Layout Transformer for Structured Document Understanding

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 7747 - 7757 May 22-27, 2022 c⃝2022 Association for Computational Linguistics LiLT: A Simple yet Effective Language-Independent Layout Transformer for Structured Document Understanding Jiapeng Wang1 Lianwen Jin∗1,3,4 Kai Ding∗2,3 1South China University of Technology, Guangzhou, China 2IntSig Information Co., Ltd, Shanghai, China 3INTSIG-SCUT Joint Laboratory of Document Recognition and Understanding, China 4Peng Cheng Laboratory, Shenzhen, China 1eejpwang@mail.scut.edu.cn, eelwjin@scut.edu.cn 2danny_ding@intsig.net Abstract Structured document understanding has at- tracted considerable attention and made sig- nificant progress recently, owing to its cru- cial role in intelligent document processing. However, most existing related models can only deal with the document data of specific language(s) (typically English) included in the pre-training collection, which is extremely limited. To address this issue, we propose a simple yet effective Language-independent Layout Transformer (LiLT) for structured doc- ument understanding. LiLT can be pre-trained on the structured documents of a single lan- guage and then directly fine-tuned on other languages with the corresponding off-the-shelf monolingual/multilingual pre-trained textual models. Experimental results on eight lan- guages have shown that LiLT can achieve com- petitive or even superior performance on di- verse widely-used downstream benchmarks, which enables language-independent benefit from the pre-training of document layout struc- ture. Code and model are publicly available at https://github.com/jpWang/LiLT. 1 Introduction Structured document understanding (SDU) aims at reading and analyzing the textual and structured information contained in scanned/digital-born doc- uments. With the acceleration of the digitization process, it has been regarded as a crucial part of intelligent document processing and required by many real-world applications in various industries such as finance, medical treatment and insurance. Recently, inspired by the rapid development of pre-trained language models of plain texts (Devlin et al., 2019; Liu et al., 2019b; Bao et al., 2020; Chi et al., 2021), many researches on structured document pre-training (Xu et al., 2020, 2021a,b; Li et al., 2021a,b,c; Appalaraju et al., 2021) have also ∗Corresponding author. (a) A form. (b) A receipt. Figure 1: The substitution of language does not appear obviously unnatural when the layout structure remains unchanged, as shown in a (a) form/(b) receipt. The detailed content has been re-synthesized to avoid the sensitive information leak. Best viewed in zoomed-in. pushed the limit of a variety of SDU tasks. How- ever, almost all of them only focus on pre-training and fine-tuning on the documents in a single lan- guage, typically English. This is extremely limited for other languages, especially in the case of lack- ing pre-training structured document data. In this regard, we consider how to make the SDU tasks enjoy language-independent benefit from the pre-training of document layout structure. Here, we give an observation as shown in Figure 1. When the layout structure remains unchanged, the substi- tution of language does not make obvious unnatu- ralness. It fully motivates us to decouple and reuse the layout invariance among different languages. Based on this inspiration, in this paper, we pro- pose a simple yet effective Language-independent Layout Transformer …

Content truncated to 500 words