Form2Seq : A Framework for Higher-Order Form Structure Extraction

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 3830–3840, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 3830 Form2Seq : A Framework for Higher-Order Form Structure Extraction Milan Aggarwal1, Hiresh Gupta2, Mausoom Sarkar1, Balaji Krishnamurthy1 Media and Data Science Research Labs, Adobe1 Adobe Experience Cloud2 Abstract Document structure extraction has been a widely researched area for decades with recent works performing it as a semantic segmenta- tion task over document images using fully- convolution networks. Such methods are lim- ited by image resolution due to which they fail to disambiguate structures in dense regions which appear commonly in forms. To mitigate this, we propose Form2Seq, a novel sequence- to-sequence (Seq2Seq) inspired framework for structure extraction using text, with a specific focus on forms, which leverages relative spa- tial arrangement of structures. We discuss two tasks; 1) Classification of low-level constituent elements (TextBlock and empty fillable Wid- get) into ten types such as field captions, list items, and others; 2) Grouping lower-level el- ements into higher-order constructs, such as Text Fields, ChoiceFields and ChoiceGroups, used as information collection mechanism in forms. To achieve this, we arrange the con- stituent elements linearly in natural reading or- der, feed their spatial and textual representa- tions to Seq2Seq framework, which sequen- tially outputs prediction of each element de- pending on the final task. We modify Seq2Seq for grouping task and discuss improvements obtained through cascaded end-to-end training of two tasks versus training in isolation. Ex- perimental results show the effectiveness of our text-based approach achieving an accu- racy of 90% on classification task and an F1 of 75.82, 86.01, 61.63 on groups discussed above respectively, outperforming segmenta- tion baselines. Further we show our frame- work achieves state of the results for table structure recognition on ICDAR 2013 dataset. 1 Introduction Various works (Hao et al., 2016; He et al., 2017; Wick and Puppe, 2018; Yang et al., 2017) have stud- ied semantic structure extraction for documents. Structure extraction is necessary for digitizing doc- uments to make them re-flowable and index-able, which is useful in web-based services (Alam and Rahman, 2003; Gupta et al., 2007; Khemakhem et al., 2018; Rahman and Alam, 2003). In this work, we look at a complex class of documents i.e., Forms that are used to capture user data by organi- zations across various domains such as government services, finance, administration, and healthcare. Such industries that have been using paper or PDF forms would want to convert them into an appro- priate digitized version (Rahman and Alam, 2003) (such as an HTML). Once these forms are made re-flowable, they can be made available across de- vices with different form factors(Alam and Rah- man, 2003; Gupta et al., 2007). This facilitates providing better form filling experiences and in- creases the ease of doing business since their users can interact with forms more conveniently and en- ables other capabilities like improved handling of filled data, applying validation checks on data filled in fields, consistent form design control1. To …

Content truncated to 500 words