HIBERT: Document Level Pre-training of Hierarchical Bidirectional Transformers for Document Summarization

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5059–5069 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 5059 HIBERT: Document Level Pre-training of Hierarchical Bidirectional Transformers for Document Summarization Xingxing Zhang, Furu Wei and Ming Zhou Microsoft Research Asia, Beijing, China {xizhang,fuwei,mingzhou}@microsoft.com Abstract Neural extractive summarization models usu- ally employ a hierarchical encoder for doc- ument encoding and they are trained us- ing sentence-level labels, which are created heuristically using rule-based methods. Train- ing the hierarchical encoder with these inac- curate labels is challenging. Inspired by the recent work on pre-training transformer sen- tence encoders (Devlin et al., 2018), we pro- pose HIBERT (as shorthand for HIerachical Bidirectional Encoder Representations from Transformers) for document encoding and a method to pre-train it using unlabeled data. We apply the pre-trained HIBERT to our summa- rization model and it outperforms its randomly initialized counterpart by 1.25 ROUGE on the CNN/Dailymail dataset and by 2.0 ROUGE on a version of New York Times dataset. We also achieve the state-of-the-art performance on these two datasets. 1 Introduction Automatic document summarization is the task of rewriting a document into its shorter form while still retaining its important content. Over the years, many paradigms for document summariza- tion have been explored (see Nenkova and McK- eown (2011) for an overview). The most popular two among them are extractive approaches and ab- stractive approaches. As the name implies, extrac- tive approaches generate summaries by extract- ing parts of the original document (usually sen- tences), while abstractive methods may generate new words or phrases which are not in the original document. Extractive summarization is usually modeled as a sentence ranking problem with length con- straints (e.g., max number of words or sentences). Top ranked sentences (under constraints) are se- lected as summaries. Early attempts mostly lever- age manually engineered features (Filatova and Hatzivassiloglou, 2004a). Based on these sparse features, sentence are selected using a classifier or a regression model. Later, the feature engineering part in this paradigm is replaced with neural net- works. Cheng and Lapata (2016) propose a hierar- chical long short-term memory network (LSTM; Hochreiter and Schmidhuber 1997) to encode a document and then use another LSTM to predict binary labels for each sentence in the document. This architecture is widely adopted recently (Nal- lapati et al., 2017; Narayan et al., 2018; Zhang et al., 2018). Our model also employs a hierarchi- cal document encoder, but we adopt a hierarchical transformer (Vaswani et al., 2017) rather a hier- archical LSTM. Because recent studies (Vaswani et al., 2017; Devlin et al., 2018) show the trans- former model performs better than LSTM in many tasks. Abstractive models do not attract much atten- tion until recently. They are mostly based on se- quence to sequence (seq2seq) models (Bahdanau et al., 2015), where a document is viewed a se- quence and its summary is viewed as another se- quence. Although seq2seq based summarizers can be equipped with copy mechanism (Gu et …

Content truncated to 500 words