IELM: An Open Information Extraction Benchmark for Pre-Trained Language Models

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 8417 - 8437 December 7-11, 2022 ©2022 Association for Computational Linguistics IELM: An Open Information Extraction Benchmark for Pre-Trained Language Models Chenguang Wang1, Xiao Liu2, Dawn Song3 1Washington University in St. Louis, 2Tsinghua University, 3UC Berkeley chenguangwang@wustl.edu, liuxiao21@mails.tsinghua.edu.cn, dawnsong@berkeley.edu Abstract We introduce a new open information extrac- tion (OIE) benchmark for pre-trained language models (LM). Recent studies have demon- strated that pre-trained LMs, such as BERT and GPT, may store linguistic and relational knowledge. In particular, LMs are able to an- swer “fill-in-the-blank” questions when given a pre-defined relation category. Instead of fo- cusing on pre-defined relations, we create an OIE benchmark aiming to fully examine the open relational information present in the pre- trained LMs. We accomplish this by turning pre-trained LMs into zero-shot OIE systems. Surprisingly, pre-trained LMs are able to ob- tain competitive performance on both standard OIE datasets (CaRB and Re-OIE2016) and two new large-scale factual OIE datasets (TAC KBP-OIE and Wikidata-OIE) that we establish via distant supervision. For instance, the zero- shot pre-trained LMs outperform the F1 score of the state-of-the-art supervised OIE methods on our factual OIE datasets without needing to use any training sets.1 1 Introduction Pre-trained language models (LM), such as BERT (Devlin et al., 2018) and GPT-3 (Brown et al., 2020), have revolutionized NLP over the last several years and advanced the state-of-the- art results in a wide set of downstream NLP tasks. Recent studies show that a considerable amount of linguistic (Hewitt and Manning, 2019; Clark et al., 2019) and relational knowledge (Petroni et al., 2019; Talmor et al., 2019; Jiang et al., 2020; Petroni et al., 2020) has been captured by the pre- trained LMs via pre-training on large-scale textual corpora. These approaches often design “fill-in-the- blank” questions based on pre-defined relations. For example, a question “Bob Dylan was born in 1Our code and datasets are available at https://github. com/cgraywang/IELM. _” is manually created for the LMs to answer the “birthplace” relation of “Bob Dylan”. Most existing approaches that evaluate what pre- trained LMs have learned are based on benchmarks with pre-defined relation categories. Yet, the bench- marks present two limitations. First, most bench- marks only cover a limited number of pre-defined relations. Therefore, it is unclear whether the pre- trained LMs have stored general open relation in- formation. For example, the Google-RE in LAMA benchmark (Petroni et al., 2019) includes only three relations (i.e., “birthplace”, “birthdate”, and “death- place”), while there are hundreds of relations avail- able in the real world scenario. Second, a major- ity of benchmarks evaluate LMs in a close man- ner. This means that the gold relation is given to the models. For example, “was born in” is given as the model’s input. Besides, the existing bench- marks often provide a single gold relation for each input sentence. However, an input sentence may indicate multiple relations, e.g., containing both “birthplace” and “birthdate” information about an argument or entity. We are curious: …

Content truncated to 500 words