SLM: Learning a Discourse Language Representation with Sentence Unshuffling

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 1551–1562, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 1551 SLM: Learning a Discourse Language Representation with Sentence Unshuffling Haejun Lee♥ Drew A. Hudson♣ Kangwook Lee♥ Christopher D. Manning♣ ♥ Samsung Research ♣ Stanford University {haejun82.lee, kw.brian.lee}@samsung.com {dorarad, manning}@cs.stanford.edu Abstract We introduce Sentence-level Language Modeling, a new pre-training objective for learning a discourse language representation in a fully self-supervised manner. Recent pre-training methods in NLP focus on learning either bottom or top-level language represen- tations: contextualized word representations derived from language model objectives at one extreme and a whole sequence representation learned by order classification of two given textual segments at the other. However, these models are not directly encouraged to capture representations of intermediate-size structures that exist in natural languages such as sentences and the relationships among them. To that end, we propose a new approach to encourage learning of a contextualized sentence-level representation by shuffling the sequence of input sentences and training a hierarchical transformer model to reconstruct the original ordering. Through experiments on downstream tasks such as GLUE, SQuAD, and DiscoEval, we show that this feature of our model improves the performance of the original BERT by large margins. 1 Introduction Recent representation learning methods in NLP such as BERT (Devlin et al., 2019) have focused on learning two types of representations: the bottom- level – a contextual representation centered at a sin- gle word, trained by recovering randomly masked tokens or predicting previous and next words, and the top-level text, implicitly represented as a single [CLS] symbol and trained by predicting a relation between input segments, which usually consist of multiple sentences. However, natural language text has, in contrast, a very dominant hierarchical struc- ture, with words grouped together into intermediate semantic units such as phrases and then sentences to convey the full meaning of a given text. Neither Figure 1: An example of a shuffled conversation and potential features that can aid in reconstructing the orig- inal sentence ordering, C → D → B → A. the transformer architecture nor the pre-training task leverages this hierarchy, treating the language as a flat sequence of tokens instead. Inspired by prior works about sentence-based representations for recurrent networks (Kiros et al., 2015; Hill et al., 2016; Gan et al., 2017; Jernite et al., 2017; Logeswaran et al., 2018; Gong et al., 2016; Chen et al., 2016), we seek to incorporate a more explicit hierarchy into the transformer by extending it with the capacity to learn contextu- alized sentence-level representations. Equipping the model with such a capacity allows it to learn languages at multiple levels of granularity, rang- ing from fine-grain connections across words to high-level discourse relations between sentences and paragraphs. Predicting an original sequence of sentences re- quires deep understanding of natural language, in- cluding a variety of phenomena such as discourse relations, coreference, temporal dependencies, en- tailments, narratives, and so on. Figure 1 shows a set of statements from a conversation in …

Content truncated to 500 words