Pretrained Language Models for Sequential Sentence Classification

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 3693–3699, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 3693 Pretrained Language Models for Sequential Sentence Classification Arman Cohan 1⋆ Iz Beltagy 1⋆ Daniel King 1 Bhavana Dalvi 1 Daniel S. Weld 1,2 1Allen Institute for Artificial Intelligence, Seattle, WA 2Allen School of Computer Science & Engineering, University of Washington, Seattle, WA {armanc,beltagy,daniel,bhavanad,danw}@allenai.org Abstract As a step toward better document-level un- derstanding, we explore classification of a se- quence of sentences into their corresponding categories, a task that requires understanding sentences in context of the document. Recent successful models for this task have used hier- archical models to contextualize sentence rep- resentations, and Conditional Random Fields (CRFs) to incorporate dependencies between subsequent labels. In this work, we show that pretrained language models, BERT (Devlin et al., 2018) in particular, can be used for this task to capture contextual dependencies with- out the need for hierarchical encoding nor a CRF. Specifically, we construct a joint sen- tence representation that allows BERT Trans- former layers to directly utilize contextual in- formation from all words in all sentences. Our approach achieves state-of-the-art results on four datasets, including a new dataset of struc- tured scientific abstracts. 1 Introduction Inspired by the importance of document-level natural language understanding, we explore clas- sification of a sequence of sentences into their respective roles or functions. For example, one might classify sentences of scientific abstracts according to rhetorical roles (e.g., Introduction, Method, Result, Conclusion, etc). We refer to this task as Sequential Sentence Classification (SSC), because the meaning of a sentence in a document is often informed by context from neighboring sentences. Recently, there have been a surge of new models for contextualized language representation, result- ing in substantial improvements on many natural language processing tasks. These models use mul- tiple layers of LSTMs (Hochreiter and Schmidhu- ⋆Equal contribution. ber, 1997) or Transformers (Vaswani et al., 2017), and are pretrained on unsupervised text with lan- guage modeling objectives such as next word pre- diction (Peters et al., 2018; Radford et al., 2018) or masked token prediction (Devlin et al., 2018; Dong et al., 2019). BERT is among the most suc- cessful models for many token- and sentence-level tasks (Devlin et al., 2018; Liu et al., 2019). In addition to a masked token objective, BERT opti- mizes for next sentence prediction, allowing it to capture sentential context. These objectives allow BERT to learn some document-level context through pretraining. In this work we explore the use of BERT for SSC. For this task, prior models are primarily based on hier- archical encoders over both words and sentences, often using a Conditional Random Field (CRF) (Lafferty et al., 2001) layer to capture document- level context (Cheng and Lapata, 2016; Jin and Szolovits, 2018; Chang et al., 2019). These mod- els encode and contextualize sentences in two con- secutive steps. In contrast, we propose an in- …

Content truncated to 500 words