MatCha: Enhancing Visual Language Pretraining with Math Reasoning and Chart Derendering

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 12756–12770 July 9-14, 2023 ©2023 Association for Computational Linguistics MATCHA : Enhancing Visual Language Pretraining with Math Reasoning and Chart Derendering Fangyu Liu♠♣∗ Francesco Piccinno♣ Syrine Krichene♣ Chenxi Pang♣ Kenton Lee♣ Mandar Joshi♣ Yasemin Altun♣ Nigel Collier♠ Julian Martin Eisenschlos♣ ♣Google DeepMind ♠University of Cambridge Abstract Visual language data such as plots, charts, and infographics are ubiquitous in the hu- man world. However, state-of-the-art vision- language models do not perform well on these data. We propose MATCHA (Math reasoning and Chart derendering pretraining) to enhance visual language models’ capabili- ties in jointly modeling charts/plots and lan- guage data. Specifically we propose several pretraining tasks that cover plot deconstruc- tion and numerical reasoning which are the key capabilities in visual language modeling. We perform the MATCHA pretraining starting from Pix2Struct, a recently proposed image- to-text visual language model. On standard benchmarks such as PlotQA and ChartQA, the MATCHA model outperforms state-of-the-art methods by as much as nearly 20%. We also examine how well the MATCHA pretraining transfers to domains such as screenshots, text- book diagrams, and document figures and ob- serve overall improvement, verifying the use- fulness of MATCHA pretraining on broader vi- sual language tasks.12 1 Introduction Visual language is the system that uses tightly inte- grated textual and visual elements to convey mean- ing (Horn, 1998). It is ubiquitous in the human world with typical examples being charts, plots and diagrams existing in places such as textbooks, sci- entific papers web pages and many more. Visual language is also highly complex – besides texts, its structural units can include line, shape, color, orientation, scale, angle, space, etc. One needs to recognize patterns from these structural units, and perform spatial grouping and/or alignment to extract information for reasoning. ∗Work done during Google internship. 1Code and models: github.com/google-research/google- research/tree/master/deplot 2For questions paper please contact fl399@cam.ac.uk and eisenjulian@google.com. Whilst being prevalent and important, there is little research on visual language understanding from the machine learning community. Vision- language models pretrained on natural images or image-text pairs crawled from the web perform badly on visual language tasks such as ChartQA (Masry et al., 2022) and PlotQA (Methani et al., 2020) due to the high complexity of jointly model- ing language and symbols (more evidence in exper- iments). Pix2Struct (Lee et al., 2023) is a recently proposed pretraining strategy for visually-situated language that significantly outperforms standard vision-language models, and also a wide range of OCR-based pipeline approaches. Pix2Struct de- signs a novel masked webpage screenshot pars- ing task and also a variable-resolution input repre- sentation for pretraining an image-to-text encode- decoder Transformer (Vaswani et al., 2017). In this work, we use Pix2Struct as the base model and further pretrain it with chart derendering and math reasoning tasks. We argue that visual language understanding needs two key ingredients: (1) layout understand- ing (including number extraction and their orga- nizations) and (2) mathematical reasoning. (1) is required to discover …

Content truncated to 500 words