Weakly-Supervised Temporal Article Grounding

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 964 - 975 December 7-11, 2022 ©2022 Association for Computational Linguistics M2D2: A Massively Multi-Domain Language Modeling Dataset Machel Reid1∗, Victor Zhong2, Suchin Gururangan2, Luke Zettlemoyer2 1The University of Tokyo, 2University of Washington machelreid@google.com, {vzhong,sg01,lsz}@cs.washington.edu Abstract We present M2D2, a fine-grained, massively multi-domain corpus for studying domain adaptation in language models (LMs). M2D2 consists of 8.5B tokens and spans 145 do- mains extracted from Wikipedia and Seman- tic Scholar. Using ontologies derived from Wikipedia and ArXiv categories, we organize the domains in each data source into 22 groups. This two-level hierarchy enables the study of relationships between domains and their ef- fects on in- and out-of-domain performance af- ter adaptation. We also present a number of in- sights into the nature of effective domain adap- tation in LMs, as examples of the new types of studies M2D2 enables. To improve in-domain performance, we show the benefits of adapt- ing the LM along a domain hierarchy; adapt- ing to smaller amounts of fine-grained domain- specific data can lead to larger in-domain per- formance gains than larger amounts of weakly relevant data. We further demonstrate a trade- off between in-domain specialization and out- of-domain generalization within and across on- tologies, as well as a strong correlation be- tween out-of-domain performance and lexical overlap between domains.1 1 Introduction Even though they can contain a wide variety of different types of domains, the texts that make up the corpora used to train and evaluate language models (LMs) are often treated as if they are all the same. This makes it challenging to characterize LM performance under diverse data distributions and understand how to effectively adapt LMs to new ones. To address these challenges, we develop M2D2, a Massively Multi-Domain Dataset, with 145 subdomains and a human-curated hierarchy for studying fine-grained domain adaptation. ∗Currently at Google Research 1We release our dataset publicly at https://github. com/machelreid/m2d2. Society and social sciences Human activites Technology and applied sciences Culture and the arts History and events Natural and physical sciences Philosophy and thinking Health and fitness Religion and belief systems General referece Mathematics and logic Society Human activities Impact of human activity Agriculture Engineering Computing Transport Culture and Humanities Sports and Recreation Visual arts Performing arts Games and Toys Mass media By period By continent By region Biology Physical sciences Nature Earth sciences Philosophy Thinking Public health Self care Exercise Health science Human medicine Nutrition Allah Major beliefs of the world Belief systems Further research tools and topics Reference works Mathematics Logic Fields of mathematics Figure 1: Visualization of the two-level fine-grained do- main hierarchy in the Wikipedia portion of M2D2. Prior work on domain transfer focuses on a small number of broad domains (typically 4-20; Guru- rangan et al., 2020; Gao et al., 2021; Gururangan et al., 2021). In contrast, domains in M2D2 are fine-grained and organized into a hierarchy derived from human-curated ontologies in Wikipedia (Fig- ure 1) and Semantic Scholar (Figure 2). Unlike …

Content truncated to 500 words