The Impact of Preprocessing on Arabic-English Statistical and Neural Machine Translation

The Impact of Preprocessing on Arabic-English Statistical and Neural Machine Translation Mai Oudah, Amjad Almahairi† and Nizar Habash Computational Approaches to Modeling Language Lab New York University Abu Dhabi, UAE †Element AI, Canada {mai.oudah,nizar.habash}@nyu.edu amjad.almahairi@elementai.com Abstract Neural networks have become the state- of-the-art approach for machine transla- tion (MT) in many languages. While linguistically-motivated tokenization tech- niques were shown to have significant ef- fects on the performance of statistical MT, it remains unclear if those techniques are well suited for neural MT. In this pa- per, we systematically compare neural and statistical MT models for Arabic-English translation on data preprecossed by vari- ous prominent tokenization schemes. Fur- thermore, we consider a range of data and vocabulary sizes and compare their effect on both approaches. Our empirical re- sults show that the best choice of tokeniza- tion scheme is largely based on the type of model and the size of data. We also show that we can gain significant improvements using a system selection that combines the output from neural and statistical MT. 1 Introduction Neural machine translation (NMT) has been rapidly attracting the attention of the research com- munity for its promising results (Cho et al., 2014b; Bahdanau et al., 2014; Wu et al., 2016; Vaswani et al., 2017). NMT is composed of two neural net- works, an encoder and a decoder, where the en- coder is fed a sentence from the source language and the decoder generates its translation, word by word, in the target language. Recently, NMT c⃝ 2019 The authors. This article is licensed under a Creative Commons 4.0 licence, no derivative works, attribution, CC- BY-ND. has been shown to outperform other MT systems in many language pairs, e.g. German-English, French-English and Basque-English (Escolano et al., 2017; Dahlmann et al., 2017; Unanue et al., 2018). While Arabic MT has been mostly devel- oped under statistical MT (SMT), NMT has also been applied and studied recently (Habash and Sa- dat, 2006; Almahairi et al., 2016; Durrani et al., 2017). Linguistically-motivated tokenization has shown to have a significant effect on SMT, particularly in the case of morphologically rich languages like Arabic (Habash and Sadat, 2006). However, it remains unclear if such techniques are well suited for NMT, where language-agnostic tokenizations, e.g. byte-pair encoding (BPE) (Sennrich et al., 2016), are widely used. Almahairi et al. (2016) has looked into Arabic SMT and NMT, achieving the highest accuracy using the Penn Arabic Treebank (ATB) tokenization, with 51.2 and 49.7 BLEU points for SMT and NMT, respectively. In this paper, we study the impact of differ- ent preprocessing techniques in Arabic-English MT on both SMT and NMT, by examining vari- ous prominent tokenization schemes. We conduct learning curve experiments to study the interac- tion between data size and the choice of tokeniza- tion scheme. We study the performance under morphology-based and frequency-based tokeniza- tion schemes, provided by MADAMIRA (Pasha et al., 2014) and BPE, respectively, on in-domain data. In addition, we evaluate the best performing models on out-of-domain data. Our results show …

Content truncated to 500 words