Improving Low-Resource NMT through Relevance Based Linguistic Features Incorporation

Proceedings of the 28th International Conference on Computational Linguistics, pages 4263–4274 Barcelona, Spain (Online), December 8-13, 2020 4263 Improving Low-Resource NMT through Relevance Based Linguistic Features Incorporation Abhisek Chakrabarty Raj Dabre Chenchen Ding Masao Utiyama Eiichiro Sumita National Institute of Information and Communications Technology 3-5 Hikaridai, Seika-cho, Soraku-gun, Kyoto, 619-0289, Japan {abhisek.chakra,raj.dabre,chenchen.ding, mutiyama,eiichiro.sumita}@nict.go.jp Abstract In this study, linguistic knowledge at different levels are incorporated into the neural machine translation (NMT) framework to improve translation quality for language pairs with extremely limited data. Integrating manually designed or automatically extracted features into the NMT framework is known to be beneficial. However, this study emphasizes that the relevance of the features is crucial to the performance. Specifically, we propose two methods, 1) self relevance and 2) word-based relevance, to improve the representation of features for NMT. Experiments are conducted on translation tasks from English to eight Asian languages, with no more than twenty thousand sentences for training. The proposed methods improve translation quality for all tasks by up to 3.09 BLEU points. Discussions with visualization provide the explainability of the proposed methods where we show that the relevance methods provide weights to features thereby enhancing their impact on low-resource machine translation. 1 Introduction Neural machine translation (NMT) (Bahdanau et al., 2015; Luong et al., 2015; Vaswani et al., 2017; Ki- taev et al., 2020) is known to give state-of-the-art translation quality for language pairs having abundance of parallel corpora. In case of resource poor scenarios, additional translation knowledge is acquired either through transfer learning in the form of pre-trained model parameters or by supplying external monolin- gual corpora. However, exploiting linguistic information effectively in low-resource conditions is still an under-researched field. Annotating the source side with various syntactic features e.g. part-of-speech (POS), lemma, dependency labels etc. can help in accurate translation when either a surface word is polysemous i.e. having multiple senses on varying context or the same word form is shared by different root words due to the inflectional nature of the language. Hence, for morphologically rich languages, ideally, the use of language specific knowledge should improve the translation quality. To the best of our knowledge, there are not an adequate amount of research works on effectively in- corporating arbitrary syntactic information into NMT. One possible reason could be that in high resource scenario the network learns from the large amount of training data to handle the problem caused by polysemy and morphological variants. In this direction, the notable works are done by (Sennrich and Haddow, 2016; Hoang et al., 2016; Li et al., 2018). Sennrich and Haddow (2016) incorporated several features at the source side by employing a separate embedding matrix for each component of a source token including the word and its associated features. Finally, all embeddings are concatenated to enrich the representation. Inspired by this work, Hoang et al. (2016) developed a method to process feature sequences of the source sentence by separate recurrent neural networks (RNNs) and combined the out- put of all RNNs using a hybrid global-local …

Content truncated to 500 words