Attention Transformer Model for Translation of Similar Languages

Proceedings of the 5th Conference on Machine Translation (WMT), pages 387–392 Online, November 19–20, 2020. c⃝2020 Association for Computational Linguistics 387 Attention Transformer Model for Translation of Similar Languages Farhan National University of Computer and Emerging Sciences, Karachi Campus k180900@nu.edu.pk Muhammad Rafi National University of Computer and Emerging Sciences, Karachi Campus muhammad.rafi@nu.edu.pk Abstract This paper illustrates our approach to the shared task on similar language translation in the fifth conference on machine translation (WMT-20). Our motivation comes from the latest state of the art neural machine transla- tion in which Transformers and Recurrent At- tention models are effectively used. A typi- cal sequence-sequence architecture consists of an encoder and a decoder Recurrent Neural Network (RNN). The encoder recursively pro- cesses a source sequence and reduces it into a fixed-length vector (context), and the decoder generates a target sequence, token by token, conditioned on the same context. In contrast, the advantage of transformers is to reduce the training time by offering a higher degree of parallelism at the cost of freedom for sequen- tial order. With the introduction of Recurrent Attention, it allows the decoder to focus effec- tively on order of the source sequence at dif- ferent decoding steps. In our approach, we have combined the recurrence based layered encoder-decoder model with the Transformer model. Our Attention Transformer model en- joys the benefits of both Recurrent Attention and Transformer to quickly learn the most probable sequence for decoding in the target language. The architecture is especially suited for similar languages (languages coming from the same family). We have submitted our sys- tem for both Indo-Aryan Language forward (Hindi to Marathi) and reverse (Marathi to Hindi) pair. Our system trains on the paral- lel corpus of the training dataset provided by the organizers and achieved an average BLEU point of 3.68 with 97.64 TER score for the Hindi-Marathi, along with 9.02 BLEU point and 88.6 TER score for Marathi-Hindi testing set. 1 Introduction This paper focuses on establishing a neural ma- chine translation model using Encoder-Decoder ar- chitecture to translate between Hindi-Marathi sen- tence pairs. We are utilizing an attention mecha- nism approach by employing the combination of re- currence based RNN Encoder-Decoder layers (Cho and et al., 2014) and latest machine translation tech- nique the Transformers (Vaswani et al., 2017) to ad- dress the complexity of WMT-20 similar language data-set (Hindi-Marathi). The motivation behind combining RNN and Transformer architecture is taken from the paper (Huang et al., 2020) to utilize the benefits of both. The formulated machine trans- lation system has numerous applications. It can be used in advertising campaigns to reach native users. The media industry can employ this technology for generating subtitles and broadcasting multilingual news to cover a wide range of native subscribers of a region. Moreover, these systems can provide native vernaculars in social media to increase the activities of indigenous individuals of a native lan- guage. Additionally, Search engines can also adopt this method to display relevant results to clients in their region-specific native language. …

Content truncated to 500 words