Better Neural Machine Translation by Extracting Linguistic Information from BERT

Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics, pages 2772–2783 April 19 - 23, 2021. ©2021 Association for Computational Linguistics 2772 Better Neural Machine Translation by Extracting Linguistic Information from BERT Hassan S. Shavarani Anoop Sarkar School of Computing Science Simon Fraser University BC, Canada {sshavara,anoop}@sfu.ca Abstract Adding linguistic information (syntax or se- mantics) to neural machine translation (NMT) has mostly focused on using point estimates from pre-trained models. Directly using the ca- pacity of massive pre-trained contextual word embedding models such as BERT (Devlin et al., 2019) has been marginally useful in NMT because effective fine-tuning is difficult to obtain for NMT without making training brittle and unreliable. We augment NMT by extracting dense fine-tuned vector-based lin- guistic information from BERT instead of us- ing point estimates. Experimental results show that our method of incorporating linguistic in- formation helps NMT to generalize better in a variety of training contexts and is no more dif- ficult to train than conventional Transformer- based NMT. 1 Introduction Probing studies into large contextual word em- beddings such as BERT (Devlin et al., 2019) have shown that these deep multi-layer models es- sentially reconstruct the traditional NLP pipeline capturing syntax and semantics (Jawahar et al., 2019); information such as part-of-speech tags, constituents, dependencies, semantic roles, co- reference resolution information (Tenney et al., 2019a,b) and subject-verb agreement information can be reconstructed from BERT embeddings (Goldberg, 2019). In this work, we wish to ex- tract the relevant pieces of linguistic information related to various levels of syntax from BERT in the form of dense vectors and then use these vectors as linguistic “experts” that neural machine translation (NMT) models can consult during translation. But can syntax help improving NMT? Linzen et al. (2016); Kuncoro et al. (2018); Sundararaman et al. (2019) have reported that learning grammati- cal structure of sentences can lead to higher levels of performance in NLP models. In particular, Sen- nrich and Haddow (2016) show that augmenting NMT models with explicit linguistic annotations improves translation quality. BERT embeddings have been previously con- sidered for improving NMT models. Clinchant et al. (2019) replace the encoder token embedding layer in a Transformer NMT model with BERT contextual embeddings. They also experiment with initializing all the encoder layers of the translation model with BERT parameters, in which case they report results on both freezing and fine-tuning the encoder parameters during training. In their ex- periments BERT embeddings can help with noisy inputs to the NMT model, but otherwise do not help improving NMT performance. Imamura and Sumita (2019) suggest that replac- ing the encoder layer with BERT embeddings and fine-tuning BERT while training the decoder leads to a catastrophic forgetting phenomenon where useful information in BERT is lost due to the mag- nitude and number of updates necessary for train- ing the translation decoder and fine-tuning BERT. They present a two-step optimization regime in which the first step freezes the BERT parameters and trains only the decoder while the …

Content truncated to 500 words