BERTology for Machine Translation: What BERT Knows about Linguistic Difficulties for Translation

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 6674–6690 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 6674 BERTology for Machine Translation: What BERT Knows about Linguistic Difficulties for Translation Yuqian Dai, Marc de Kamps, Serge Sharoff Centre for Translation Studies, University of Leeds, LS2 9JT, United Kingdom {mlyd, m.dekamps, s.sharoff}@leeds.ac.uk Abstract Pre-trained transformer-based models, such as BERT, have shown excellent performance in most natural language processing benchmark tests, but we still lack a good understanding of the linguistic knowledge of BERT in Neural Machine Translation (NMT). Our work uses syntactic probes and Quality Estimation (QE) models to analyze the performance of BERT’s syntactic dependencies and their impact on machine translation quality, exploring what kind of syntactic dependencies are difficult for NMT engines based on BERT. While our probing experiments confirm that pre-trained BERT “knows” about syntactic dependencies, its ability to recognize them often decreases after fine-tuning for NMT tasks. We also detect a relationship between syntactic dependencies in three languages and the quality of their translations, which shows which specific syntactic dependencies are likely to be a significant cause of low-quality translations. Keywords: Neural Machine Translation, BERT, Syntactic Dependencies, Probing Experiments, Quality Estimation 1. Introduction The encoder-decoder used in Neural Machine Trans- lation (NMT) is a distinctive representation architec- ture whose encoder and decoder are the product of representation learning for both source and target lan- guages. The Transformer model (Vaswani et al., 2017) proposed in recent years is even more efficient in de- scribing long-term links between words through a self- attention mechanism. It has become one of the most widely used models in machine translation, and the joint training of encoder and decoder makes it pos- sible to add pre-training with better generalization abil- ity to the machine translation task. Compared with the LSTM-based pre-trained model ELMo (Peters et al., 2018), the pre-trained model BERT (Devlin et al., 2019) takes the role of pre-training with word repres- entation to a new level. The auto-encoding approach proposed by BERT allows the model to be modeled with self-attention in the pre-training phase, which further enhances the representation capability of the model. Inspired by BERT, more pre-trained models are proposed, such as XLM (Conneau and Lample, 2019) and RoBERTa (Liu et al., 2019). The large-scale data learning that pre-trained models possess makes it pos- sible to acquire more general linguistic knowledge and input representations and provides better initialization parameters and generalization capabilities for down- stream tasks (Edunov et al., 2019). Given the suc- cess of BERT in language understanding tasks, much work has been done on how to incorporate BERT to improve the translation quality of machine translation (Clinchant et al., 2019; Weng et al., 2020). Although BERT can be added to either the encoder or the de- coder part, it is more common to incorporate it into a translation task as an encoder or as part of an en- coder (Imamura and Sumita, 2019; Yang et al., 2020; Zhu …

Content truncated to 500 words