LayoutReader: Pre-training of Text and Layout for Reading Order Detection

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 4735–4744 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 4735 LayoutReader: Pre-training of Text and Layout for Reading Order Detection Zilong Wang1⇤, Yiheng Xu2⇤, Lei Cui2, Jingbo Shang1, Furu Wei2 1University of California, San Diego 2Microsoft Research Asia {zlwang,jshang}@ucsd.edu {t-yihengxu,lecu,fuwei}@microsoft.com Abstract Reading order detection is the cornerstone to understanding visually-rich documents (e.g., receipts and forms). Unfortunately, no existing work took advantage of advanced deep learn- ing models because it is too laborious to anno- tate a large enough dataset. We observe that the reading order of WORD documents is em- bedded in their XML metadata; meanwhile, it is easy to convert WORD documents to PDFs or images. Therefore, in an automated man- ner, we construct ReadingBank, a benchmark dataset that contains reading order, text, and layout information for 500,000 document im- ages covering a wide spectrum of document types. This first-ever large-scale dataset un- leashes the power of deep neural networks for reading order detection. Specifically, our pro- posed LayoutReader captures the text and lay- out information for reading order prediction using the seq2seq model. It performs almost perfectly in reading order detection and signif- icantly improves both open-source and com- mercial OCR engines in ordering text lines in their results in our experiments. The dataset and models are publicly available at https: //aka.ms/layoutreader. 1 Introduction Reading order detection, aiming to capture the word sequence which can be naturally compre- hended by human readers, is a fundamental task for visually-rich document understanding. Cur- rent off-the-shelf methods usually directly borrow the results from the Optical Character Recognition (OCR) engines (Xu et al., 2020) while most OCR engines arrange the recognized tokens or text lines in a top-to-bottom and left-to-right way (Clausner et al., 2013). Apparently, as shown in Figure 1, this heuristic method is not optimal for certain docu- ment types, such as multi-column templates, forms, invoices, and many others. An incorrect reading or- der will lead to unacceptable results for document ⇤Contributions during internship at MSRA. understanding tasks such as the information extrac- tion from receipts/invoices. Therefore, an accurate reading order detection model is indispensable to the document understanding tasks. In the past decades, some conventional machine learning based or rule based methods (Aiello et al., 2003; Ceci et al., 2007; Malerba and Ceci, 2007; Malerba et al., 2008; Ferilli et al., 2014) have been proposed. However, these approaches are usually trained with only a small number of samples within a restricted domain or resort to unsupervised meth- ods with empirical rules, because it is too laborious to annotate a large enough dataset. These mod- els can barely show case studies of certain reading order scenarios and cannot be easily adapted for real-world reading order problems. Recently, deep learning models (Li et al., 2020a) have been ap- plied to address the reading order issues for images from E-commerce platforms. Although good per- formance has been achieved, it is time-consuming and labor-intensive to produce an in-house dataset, while …

Content truncated to 500 words