DeepStruct: Pretraining of Language Models for Structure Prediction

DEEPSTRUCT: Pretraining of Language Models for Structure Prediction Chenguang Wang∗†, Xiao Liu¶†, Zui Chen¶†, Haoyun Hong¶, Jie Tang¶, Dawn Song∗ ∗UC Berkeley, ¶Tsinghua University {chenguangwang,dawnsong}@berkeley.edu, jietang@tsinghua.edu.cn {liuxiao21,chenzui19,honghy17}@mails.tsinghua.edu.cn Abstract We introduce a method for improving the struc- tural understanding abilities of language mod- els. Unlike previous approaches that finetune the models with task-specific augmentation, we pretrain language models on a collection of task-agnostic corpora to generate structures from text. Our structure pretraining enables zero-shot transfer of the learned knowledge that models have about the structure tasks. We study the performance of this approach on 28 datasets, spanning 10 structure prediction tasks including open information extraction, joint en- tity and relation extraction, named entity recog- nition, relation classification, semantic role la- beling, event extraction, coreference resolution, factual probe, intent detection, and dialogue state tracking. We further enhance the pretrain- ing with the task-specific training sets. We show that a 10B parameter language model transfers non-trivially to most tasks and ob- tains state-of-the-art performance on 21 of 28 datasets that we evaluate.1 1 Introduction Pretrained language models (LMs) have revolu- tionized NLP over the last few years (Peters et al., 2018; Devlin et al., 2019; Radford et al., 2019b), increasingly adept in performing the flexible and task-agnostic downstream transfer. Their transfer performance is less studied in structure prediction tasks, however. Well-studied tasks mainly focus on understanding one particular aspect of the text, such as predicting the next word that comes after as in language modeling. Unlike those downstream tasks, structure prediction requires the structural un- derstanding of the text for further integrating multi- ple relevant aspects into a structure. For instance, a typical structure prediction task, called open in- formation extraction, seeks the entire structural in- †Equal contribution. 1The code and datasets are available at https:// github.com/cgraywang/deepstruct. Open information extraction (Iago; Born in; 1951) Joint entity and relation extraction (Iago; instance of; person) (Iago; city_of_birth; Tbilisi) Relation classification (Iago; city_of_birth; Tbilisi) Multi-task corpora (optional) input: jer conll04: An art exhibit is at the Haka Theatre (Haka Theatre; instance of; theatre) output: ner genia: Japan began the defence of their Asian Cup title input: output: Named entity recognition (Iago; instance of; person) (Asian Cup; instance of; race) (Japan; instance of; location) (art exhibit; located in; Haka Theatre) (Iago; is a; Georgian artist) Born in 1951 in Tbilisi, Iago is a Georgian artist DeepStruct input: triple: The couple have a daughter output: (couple; have; a daughter) Task-agnostic corpora ...... input: entity: He played for FIFA (Structure pretraining) input: relation: The book Fly is in English output: (Fly; language; English) output: (He; instance of; human) (FIFA; instance of; club) ...... Figure 1: Summary of our approach and results. Upper: an overview of DEEPSTRUCT and the proposed structure pretrain- ing. Lower: performance of our 10B DEEPSTRUCT zero-shot and multi-task, compared with 175B GPT-3 zero-shot. formation in a sentence (Figure 2). Different from traditional NLP tasks, structure prediction takes one step further and serves as a natural testbed for the structural understanding competence of LMs. It is non-trivial …

Content truncated to 500 words