MUSTIE: Multimodal Structural Transformer for Web Information Extraction

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 2405–2420 July 9-14, 2023 ©2023 Association for Computational Linguistics MUSTIE: Multimodal Structural Transformer for Web Information Extraction Qifan Wang1, Jingang Wang2∗, Xiaojun Quan3, Fuli Feng4, Zenglin Xu5, Shaoliang Nie1, Sinong Wang1, Madian Khabsa1, Hamed Firooz1 and Dongfang Liu6* 1Meta AI 2Meituan Lab 3Sun Yat-sen University 4University of Science and Technology of China 5Peng Cheng Lab 6Rochester Institute of Technology wqfcr@fb.com Abstract The task of web information extraction is to ex- tract target fields of an object from web pages, such as extracting the name, genre and actor from a movie page. Recent sequential model- ing approaches have achieved state-of-the-art results on web information extraction. How- ever, most of these methods only focus on ex- tracting information from textual sources while ignoring the rich information from other modal- ities such as image and web layout. In this work, we propose a novel MUltimodal Struc- tural Transformer (MUST) that incorporates multiple modalities for web information ex- traction. Concretely, we develop a structural encoder that jointly encodes the multimodal information based on the HTML structure of the web layout, where high-level DOM nodes, low-level text, and image tokens are introduced to represent the entire page. Structural atten- tion patterns are designed to learn effective cross-modal embeddings for all DOM nodes and low-level tokens. An extensive set of ex- periments has been conducted on WebSRC and Common Crawl benchmarks. Experimental re- sults demonstrate the superior performance of MUST over several state-of-the-art baselines. 1 Introduction The world wide web has grown explosively in the past decades, with millions of new web pages be- ing created everyday. Web pages and documents have been widely used and become a powerful re- source for humans to obtain information. For exam- ple, Figure 1 shows a movie page from the IMDB website, which contains structured movie infor- mation including movie name, description, genre, etc. This information is essential to facilitate new experiences in applications like web search and retrieval (Crescenzi and Mecca, 2004; Yan et al., 2009). There has been an enduring demand for au- tomatic information extraction from unstructured ∗Corresponding authors. Figure 1: An example of a movie page from the IMDB website. The extractions of movie name, description, genre, duration, director, actor and release date are high- lighted with colored bounding boxes on the web page. or semi-structured web pages to create structured knowledge bases (Chang et al., 2006; Hao et al., 2011). Therefore, it is an important research prob- lem to extract structured information from web pages (Carlson and Schafer, 2008). Web information extraction (Manabe and Tajima, 2015; Wu et al., 2018) poses a lot of challenges to researchers in both academia and industry, due to the unstructured nature and the diverse layout pat- terns of the web documents (Xiong et al., 2019; Lockard et al., 2019). Moreover, web data often contains multiple modalities such as texts, tables, and images. A substantial amount of research (Katti et al., 2018; …

Content truncated to 500 words