STAPI: An Automatic Scraper for Extracting Iterative Title-Text Structure from Web Documents

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 3461–3470 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 3461 STAPI: An Automatic Scraper for Extracting Iterative Title-Text Structure from Web Documents Nan Zhang1, Shomir Wilson1, Prasenjit Mitra1,2 1College of Information Sciences and Technology, The Pennsylvania State University, USA 2L3S Research Center, Germany {njz5124, shomir, pmitra}@psu.edu Abstract Formal documents often are organized into sections of text, each with a title, and extracting this structure remains an under-explored aspect of natural language processing. This iterative title-text structure is valuable data for building models for headline generation and section title generation, but there is no corpus that contains web documents annotated with titles and prose texts. Therefore, we propose the first title-text dataset on web documents that incorporates a wide variety of domains to facilitate downstream training. We also introduce STAPI (Section Title And Prose text Identifier), a two-step system for labeling section titles and prose text in HTML documents. To filter out unrelated content like document footers, its first step involves a filter that reads HTML documents and proposes a set of textual candidates. In the second step, a typographic classifier takes the candidates from the filter and categorizes each one into one of the three pre-defined classes (title, prose text, and miscellany). We show that STAPI significantly outperforms two baseline models in terms of title-text identification. We release our dataset along with a web application to facilitate supervised and semi-supervised training in this domain. Keywords: Iterative Title-Text Dataset, Title-Text Identification, Automatic Scraping, Information Extraction 1. Introduction Many documents (e.g., news articles, how-to guides, books, privacy policies, terms of service documents, and others) adopt an iterative title-text structure. They often separate their contents into multiple sections, and each section comes with a title. Sometimes a section is further broken into a hierarchy of subsections. The first column of Figure 1 displays typical snippets of title- text structure. This iterative title-text structure is po- tentially valuable for natural language generation tasks such as headline and section title generation, and ques- tion answering. For headline and section title genera- tion, it provides labels for whether each text segment is a title or prose text in the original document. Therefore, once a large collection of documents is identified, it can facilitate rapid construction of a large-scale section ti- tle dataset. For question answering and dialogue sys- tems, the iterative title-text structure may help a model find the target section, thus improving the model per- formance and reducing the search space. This structure may also facilitate topic extraction and sentiment anal- ysis by leveraging the titles in a document. However, extracting this iterative title-text structure from web documents is not well addressed in previ- ous research. First of all, there is no corpus that con- tains web documents annotated with titles and prose texts. Moreover, our analysis about an existing ap- proach (Gopinath et al., 2018) shows two cases where existing algorithms run into difficulties: (1) when …

Content truncated to 500 words