ERNIE-Gram: Pre-Training with Explicitly N-Gram Masked Language Modeling for Natural Language Understanding

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1702–1715 June 6–11, 2021. ©2021 Association for Computational Linguistics 1702 ERNIE-Gram: Pre-Training with Explicitly N-Gram Masked Language Modeling for Natural Language Understanding Dongling Xiao, Yu-Kun Li, Han Zhang, Yu Sun, Hao Tian, Hua Wu and Haifeng Wang Baidu Inc., China {xiaodongling,liyukun01,zhanghan17,sunyu02, tianhao,wu hua,wanghaifeng}@baidu.com Abstract Coarse-grained linguistic information, such as named entities or phrases, facilitates adequate- ly representation learning in pre-training. Pre- vious works mainly focus on extending the ob- jective of BERT’s Masked Language Model- ing (MLM) from masking individual tokens to contiguous sequences of n tokens. We ar- gue that such contiguously masking method neglects to model the intra-dependencies and inter-relation of coarse-grained linguistic infor- mation. As an alternative, we propose ERNIE- Gram, an explicitly n-gram masking method to enhance the integration of coarse-grained in- formation into pre-training. In ERNIE-Gram, n-grams are masked and predicted directly us- ing explicit n-gram identities rather than con- tiguous sequences of n tokens. Furthermore, ERNIE-Gram employs a generator model to sample plausible n-gram identities as optional n-gram masks and predict them in both coarse- grained and fine-grained manners to enable comprehensive n-gram prediction and rela- tion modeling. We pre-train ERNIE-Gram on English and Chinese text corpora and fine- tune on 19 downstream tasks. Experimental results show that ERNIE-Gram outperforms previous pre-training models like XLNet and RoBERTa by a large margin, and achieves comparable results with state-of-the-art meth- ods. The source codes and pre-trained mod- els have been released at https://github. com/PaddlePaddle/ERNIE. 1 Introduction Pre-trained on large-scaled text corpora and fine- tuned on downstream tasks, self-supervised rep- resentation models (Radford et al., 2018; Devlin et al., 2019; Liu et al., 2019; Yang et al., 2019; Lan et al., 2020; Clark et al., 2020) have achieved remarkable improvements in natural language un- derstanding (NLU). As one of the most prominent pre-trained models, BERT (Devlin et al., 2019) em- ploys masked language modeling (MLM) to learn representations by masking individual tokens and predicting them based on their bidirectional context. However, BERT’s MLM focuses on the represen- tations of fine-grained text units (e.g. words or subwords in English and characters in Chinese), rarely considering the coarse-grained linguistic in- formation (e.g. named entities or phrases in English and words in Chinese) thus incurring inadequate representation learning. Many efforts have been devoted to integrate coarse-grained semantic information by indepen- dently masking and predicting contiguous se- quences of n tokens, namely n-grams, such as named entities, phrases (Sun et al., 2019b), whole words (Cui et al., 2019) and text spans (Joshi et al., 2020). We argue that such contiguously masking strategies are less effective and reliable since the prediction of tokens in masked n-grams are inde- pendent of each other, which neglects the intra- dependencies of n-grams. Specifically, given a masked n-gram w={x1, ..., xn}, x∈VF , we max- imize p(w) = �n i=1 p(xi|c) for n-gram learning, where models learn to recover w in a huge and sparse prediction …

Content truncated to 500 words