FeatureBART: Feature Based Sequence-to-Sequence Pre-Training for Low-Resource NMT

Proceedings of the 29th International Conference on Computational Linguistics, pages 5014–5020 October 12–17, 2022. 5014 FeatureBART: Feature Based Sequence-to-Sequence Pre-Training for Low-Resource NMT Abhisek Chakrabarty∗, Raj Dabre∗, Chenchen Ding, Hideki Tanaka, Masao Utiyama and Eiichiro Sumita National Institute of Information and Communications Technology 3-5 Hikaridai, Seika-cho, Soraku-gun, Kyoto, 619-0289, Japan {abhisek.chakra,raj.dabre,chenchen.ding,hideki.tanaka, mutiyama,eiichiro.sumita}@nict.go.jp Abstract In this paper we present FeatureBART, a linguistically motivated sequence-to-sequence monolingual pre-training strategy in which syn- tactic features such as lemma, part-of-speech and dependency labels are incorporated into the span prediction based pre-training frame- work (BART). These automatically extracted features are incorporated via approaches such as concatenation and relevance mechanisms, among which the latter is known to be better than the former. When used for low-resource NMT as a downstream task, we show that these feature based models give large improvements in bilingual settings and modest ones in multi- lingual settings over their counterparts that do not use features. 1 Introduction Sequence-to-sequence (S2S) pre-training done via denoising objectives on monolingual corpora is known to improve generation quality in low- resource settings (Lewis et al., 2020). This has been extensively explored for neural machine translation, however most works show that the improvements for translation into English are more pronounced than those for translation into a non-English lan- guage (Liu et al., 2020; Tang et al., 2020). One reason for this is that pre-training leads to a strong decoder that learns to de-noise masked inputs, whereas the knowledge retained in the encoder is rather limited. Thus far, there has been no explicit effort towards improving the contribution of the encoder during pre-training. Most pre-training methods rely on the power of large models and large corpora, but ignore the possibility of incorporating linguistic knowledge into the pre-trained model. On the other hand, there are several works which show that incorpo- rating linguistic knowledge in the form of lemma, part-of-speech tags and dependency labels, lead ∗ Equal contribution. to a significant improvement in translation qual- ity (Sennrich and Haddow, 2016; Hoang et al., 2016; Li et al., 2018; Pan et al., 2020; Chakrabarty et al., 2020) in both low- and high-resource set- tings. Most recently, in an extremely low-resource setting, Chakrabarty et al. (2020) show the effec- tiveness of using the aforementioned linguistic fea- tures, especially when their influence on the model is controlled via relevance mechanisms that ap- propriately scale feature embeddings before they augment word embeddings. We hypothesize that incorporating linguistic features, into the denoising based pre-training framework, should improve the quality of pre-training, which should then have a positive impact on the translation quality via fine- tuning. To this end, we propose FeatureBART, a feature based sequence-to-sequence pre-training method. In FeatureBART, linguistic features are obtained via automatic annotators and then converted into embeddings, which are used to augment the word embeddings of the encoder. Feature embeddings, are incorporated by either naive concatenation with the word embeddings or by first weighing them with a relevance mechanism and then adding them to word embeddings. The model itself is trained using …

Content truncated to 500 words