TableVLM: Multi-modal Pre-training for Table Structure Recognition

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 2437–2449 July 9-14, 2023 ©2023 Association for Computational Linguistics TableVLM: Multi-modal Pre-training for Table Structure Recognition Leiyuan Chen1,2, Chengsong Huang1,2 Xiaoqing Zheng1,2,∗ Jinshu Lin3, Xuanjing Huang1,2 1School of Computer Science, Fudan University, Shanghai, China 2Shanghai Key Laboratory of Intelligent Information Processing 3Hundsun {20210240034,huangcs19,zhengxq}@fudan.edu.cn linjs13607@hundsun.com, xjhuang@fudan.edu.cn Abstract Tables are widely used in research and busi- ness, and are suitable for human consumption, but not easily machine-processable, particu- larly when tables are present in images. One of the main challenges to extracting data from images of tables is to accurately recognize ta- ble structures, especially for complex tables with cross rows and columns. In this study, we propose a novel multi-modal pre-training model for table structure recognition, named TableVLM. With a two-stream multi-modal transformer-based encoder-decoder architec- ture, TableVLM learns to capture rich table structure-related features by multiple carefully- designed unsupervised objectives inspired by the notion of masked visual-language model- ing. To pre-train this model, we also created a dataset, called ComplexTable, which consists of 1, 000K samples to be released publicly. Ex- periment results show that the model built on pre-trained TableVLM can improve the per- formance up to 1.97% in tree-editing-distance- score on ComplexTable. 1 Introduction Tables are quite useful for displaying data in an organized manner and they are widely used in re- search and business due to their readability and simplicity. Recently, such semi-structured (tabu- lar) data has attracted more attention because of its ubiquitous presence in almost all types of doc- uments such as medical records, insurance files, and scientific articles (Staar et al., 2018). How- ever, in many cases, we can only access images of tabular data. The format information will be lost if a table is turned into an image. It is very hard to recover the structure of tables from their images because tables differ significantly in struc- ture, notation, and representation. Once the table structure is accurately recognized, its texts can be easily extracted with the help of optical character recognition (OCR) toolkit and reorganized into a ta- (a) An example of table image with multi-column headers, multi-row headers and some missing dividing lines. (b) The ground truth structure of the example table. The table cells used to show different headers are indicated by distinct colors. (c) The structure recognized by PDFlux (d) The structure recognized by Tabby Figure 1: Some typical mistakes made by two represen- tative table recognition toolkits: PDFlux1 and Tabby2 (Shigarov et al., 2018). The former fails to recognize the multi-column header of “Parental illness type (PIT)” while the latter can not arrange all the headers as they were originally presented. ble as they were presented in the image. Therefore, table structure recognition is considered a critical task for automatic document understanding, and many competitions around this task have been held in the research and business communities (Göbel et al., 2013; Gao et al., 2019; Jimeno-Yepes et al., 2021; Kayal et al., 2021). …

Content truncated to 500 words