How much pretraining data do language models need to learn syntax?

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 133–139 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 133 Multiplex Graph Neural Network for Extractive Text Summarization Baoyu Jing†, Zeyu You‡, Tao Yang‡, Wei Fan‡ and Hanghang Tong† †University of Illinois at Urbana-Champaign ‡Tencent America {baoyuj2, htong}@illinois.edu {zeyuyou, tytaoyang, davidwfan}@tencent.com Abstract Extractive text summarization aims at extract- ing the most representative sentences from a given document as its summary. To extract a good summary from a long text document, sen- tence embedding plays an important role. Re- cent studies have leveraged graph neural net- works to capture the inter-sentential relation- ship (e.g., the discourse graph) to learn con- textual sentence embedding. However, those approaches neither consider multiple types of inter-sentential relationships (e.g., semantic similarity & natural connection), nor model intra-sentential relationships (e.g, semantic & syntactic relationship among words). To ad- dress these problems, we propose a novel Mul- tiplex Graph Convolutional Network (Multi- GCN) to jointly model different types of rela- tionships among sentences and words. Based on Multi-GCN, we propose a Multiplex Graph Summarization (Multi-GraS) model for extrac- tive text summarization. Finally, we evaluate the proposed models on the CNN/DailyMail benchmark dataset to demonstrate the effec- tiveness of our method. 1 Introduction Numerous documents from a variety of sources are uploaded to the Internet or database everyday, such as news articles (Hermann et al., 2015), sci- entific papers (Qazvinian and Radev, 2008) and electronic health records (Jing et al., 2019). How to effectively digest the overwhelming informa- tion has always been a fundamental question in natural language processing (Nenkova and McKe- own, 2011). This question has sparked the research interests in the task of extractive text summariza- tion, which aims to generate a short summary of a document by extracting the most representative sentences from it. Most of the recent methods (Cheng and Lap- ata, 2016; Narayan et al., 2018; Luo et al., 2019; Wang et al., 2020a; Mendes et al., 2019; Zhou et al., 2018) formulate the task of extractive text summa- rization as a sequence labeling task, where the la- bels indicate whether a sentence should be included in the summary. To extract sentence features, ex- isting approaches generally use Recurrent Neural Networks (RNN) (Yasunaga et al., 2017; Nallapati et al., 2017; Zhou et al., 2018; Mendes et al., 2019; Luo et al., 2019; Cheng and Lapata, 2016), Convo- lutional Neural Networks (CNN) (Cheng and La- pata, 2016; Luo et al., 2019; Narayan et al., 2018) or Transformers (Zhong et al., 2019; Liu and Lap- ata, 2019a). Endeavors have been made to develop models to capture various sentence-level relations. Early studies, such as LexRank (Radev, 2004) and TextRank (Mihalcea and Tarau, 2004), built similar- ity graphs among sentences and leverage PageRank (Page et al., 1999) to score them. Later, graph neu- ral networks e.g., Graph Convolutional Network (GCN) (Kipf and Welling, 2016) have been adopted on various inter-sentential graphs, such as the ap- proximate discourse graph (Yasunaga et al., 2017), the discourse graph …

Content truncated to 500 words