Cloze-driven Pretraining of Self-attention Networks

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 5360–5369, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 5360 Cloze-driven Pretraining of Self-attention Networks Alexei Baevski, Sergey Edunov∗, Yinhan Liu∗, Luke Zettlemoyer, Michael Auli Facebook AI Research Menlo Park, CA and Seattle, WA Abstract We present a new approach for pretraining a bi-directional transformer model that provides significant performance gains across a vari- ety of language understanding problems. Our model solves a cloze-style word reconstruction task, where each word is ablated and must be predicted given the rest of the text. Experi- ments demonstrate large performance gains on GLUE and new state of the art results on NER as well as constituency parsing benchmarks, consistent with BERT. We also present a de- tailed analysis of a number of factors that con- tribute to effective pretraining, including data domain and size, model capacity, and varia- tions on the cloze objective. 1 Introduction Language model pretraining has recently been shown to provide significant performance gains for a range of challenging language understand- ing problems (Dai and Le, 2015; Peters et al., 2018; Radford et al., 2018). However, existing work has either used unidirectional (left-to-right) language models (LMs) (Radford et al., 2018) or bi-directional (both left-to-right and right-to-left) LMs (BiLMs) where each direction is trained with an independent loss function (Peters et al., 2018). In this paper, we show that even larger perfor- mance gains are possible by jointly pretraining both directions of a large language-model-inspired self-attention cloze model. Our bi-directional transformer architecture pre- dicts every token in the training data (Figure 1). We achieve this by introducing a cloze-style train- ing objective where the model must predict the center word given left-to-right and right-to-left context representations. Our model separately computes both forward and backward states with ∗Equal contribution. <s> a b c <s> Block1 BlockN Block1 BlockN Block1 BlockN Block1 BlockN comb b Figure 1: Illustration of the model. Blocki is a standard transformer decoder block. Green blocks operate left to right by masking future time-steps and blue blocks op- erate right to left. At the top, states are combined with a standard multi-head self-attention module whose out- put is fed to a classifier that predicts the center token. a masked self-attention architecture, that closely resembles a language model. At the top of the net- work, the forward and backward states are com- bined to jointly predict the center word. This ap- proach allows us to consider both contexts when predicting words and to incur loss for every word in the training set, if the model does not assign it high likelihood. Experiments on the GLUE (Wang et al., 2018) benchmark show strong gains over the state of the art for each task, including a 9.1 point gain on RTE over Radford et al. (2018). These improvements are consistent with, if slightly behind, BERT (De- vlin et al., 2018), which we will discuss in more …

Content truncated to 500 words