WebIE: Faithful and Robust Information Extraction on the Web

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 7734–7755 July 9-14, 2023 ©2023 Association for Computational Linguistics WEBIE: Faithful and Robust Information Extraction on the Web Chenxi Whitehouse1,2,*, Clara Vania2, Alham Fikri Aji2,**, Christos Christodoulopoulos2, Andrea Pierleoni2 1City, University of London 2Amazon Alexa AI, Cambridge, UK chenxi.whitehouse@city.ac.uk {vaniclar, chrchrs, apierleoni}@amazon.co.uk Abstract Extracting structured and grounded fact triples from raw text is a fundamental task in Informa- tion Extraction (IE). Existing IE datasets are typically collected from Wikipedia articles, us- ing hyperlinks to link entities to the Wikidata knowledge base. However, models trained only on Wikipedia have limitations when applied to web domains, which often contain noisy text or text that does not have any factual informa- tion. We present WEBIE, the first large-scale, entity-linked closed IE dataset consisting of 1.6M sentences automatically collected from the English Common Crawl corpus. WEBIE also includes negative examples, i.e. sentences without fact triples, to better reflect the data on the web. We annotate ∼21K triples from WEBIE through crowdsourcing and introduce mWEBIE, a translation of the annotated set in four other languages: French, Spanish, Por- tuguese, and Hindi. We evaluate the in-domain, out-of-domain, and zero-shot cross-lingual per- formance of generative IE models and find mod- els trained on WEBIE show better generalisabil- ity. We also propose three training strategies that use entity linking as an auxiliary task. Our experiments show that adding Entity-Linking objectives improves the faithfulness of our gen- erative IE models1. 1 Introduction Information Extraction (IE) is the task of extract- ing structured information from unstructured text, usually in the form of triples <subject, relation, object>. It is essential for many Natural Language Processing applications such as knowledge base population, question answering, faithful summari- sation, and fake news detection (Trisedya et al., 2019; Huguet Cabot and Navigli, 2021; Narayan et al., 2021; Whitehouse et al., 2022). ∗ Work conducted as Research Intern at Amazon Alexa AI. ∗∗ Now at Mohamed Bin Zayed University of Artificial In- telligence (MBZUAI), Abu Dhabi, UAE. 1Dataset, code and additional materials are available at https://github.com/amazon-science/WebIE. Typically, two pieces of information are needed for training closed IE2 systems: (i) the entities mentioned in the text and (ii) the relations that ex- ist between each pair of entities. Obtaining such information requires expensive annotations, there- fore most existing IE datasets, such as WikiNRE (Trisedya et al., 2019) or REBEL (Huguet Cabot and Navigli, 2021), are built using Wikipedia, as en- tity information is available through hyperlinks and relation information can be automatically extracted via distant supervision (DS) approach (Mintz et al., 2009) using a knowledge base (KB) such as Wiki- data. The DS approach assumes that if two entities are connected through a relation in a KB, then the sentences that mention both entities together ex- press the relation. While models trained only on this fact-rich do- main3 have shown to be useful for IE applications, they have limited capacity when applied to ex- tracting information in other web domains, …

Content truncated to 500 words