BERT, mBERT, or BiBERT? A Study on Contextualized Embeddings for Neural Machine Translation

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6663–6675 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 6663 BERT, MBERT, or BIBERT? A Study on Contextualized Embeddings for Neural Machine Translation Haoran Xu, Benjamin Van Durme, Kenton Murray Johns Hopkins University {hxu64,vandurme,kenton}@jhu.edu Abstract The success of bidirectional encoders using masked language models, such as BERT, on numerous natural language processing tasks has prompted researchers to attempt to incor- porate these pre-trained models into neural machine translation (NMT) systems. How- ever, proposed methods for incorporating pre- trained models are non-trivial and mainly fo- cus on BERT, which lacks a comparison of the impact that other pre-trained models may have on translation performance. In this paper, we demonstrate that simply using the output (contextualized embeddings) of a tailored and suitable bilingual pre-trained language model (dubbed BIBERT) as the input of the NMT encoder achieves state-of-the-art translation performance. Moreover, we also propose a stochastic layer selection approach and a con- cept of dual-directional translation model to ensure the sufficient utilization of contextual- ized embeddings. In the case of without us- ing back translation, our best models achieve BLEU scores of 30.45 for En→De and 38.61 for De→En on the IWSLT’14 dataset, and 31.26 for En→De and 34.94 for De→En on the WMT’14 dataset, which exceeds all pub- lished numbers12. 1 Introduction Pre-trained language models (LMs), trained on a large-scale unlabeled data to capture rich rep- resentations of the input, such as ELMO (Peters et al., 2018), BERT (Devlin et al., 2019), XLNET (Yang et al., 2019) and XLM (Conneau and Lam- ple, 2019) have increasingly attracted attention in various NLP tasks. Either utilizing context-aware representations of input tokens (Peters et al., 2018) or fine-tuning the pre-trained parameters (Devlin 1Code is available at: https://github.com/ fe1ixxu/BiBERT. 2Our BiBERT is released at: https://huggingface. co/jhu-clsp/bibert-ende. et al., 2019) both lead to significant improvement for downstream tasks. Figure 1: The overview of methods: a series of additive improvements to the use of contextualized embeddings on IWSLT’14 dataset. Experimenting over various pre- trained language models, we show that our BIBERT, a bilingual English-German language model, vastly outperforms all other methods (Section 2). Adding stochastic layer selection to BIBERT improves perfor- mance (Section 3). Finally, innovative dual-directional training and fine-tuning with the previous two meth- ods yield around 2 BLEU point gains over the previous state-of-the-art result (Wu et al., 2021) (Section 4). Inspired by the superior performance of BERT on many other tasks, researchers have investigated leveraging using this pre-trained masked language model to enhance translation models, e.g., initializ- ing the parameters of the model’s encoder with BERT parameters (Rothe et al., 2020), and in- corporating the output of BERT to each layer of the encoder (Zhu et al., 2020; Weng et al., 2020). In this paper, we demonstrate simply using the output of a pre-trained language model as the in- put of NMT systems can achieve state-of-the-art results on IWLST’14 (Cettolo et al., 2014) and WMT’14 (Bojar et al., 2014) English↔German (En↔De) translation …

Content truncated to 500 words