Training Deployable General Domain MT for a Low Resource Language Pair: English-Bangla

Training Deployable General Domain MT for a Low Resource Language Pair: English–Bangla Sandipan Dandapat and William Lewis Microsoft AI & Research {sadandap,wilewis}@microsoft.com Abstract A large percentage of the world’s popula- tion speaks a language of the Indian sub- continent, what we will call here Indic lan- guages, comprising languages from both Indo-European (e.g., Hindi, Bangla, Gu- jarati, etc.) and Dravidian (e.g., Tamil, Tel- ugu, Malayalam, etc.) families, upwards of 1.5 Billion people. A universal char- acteristic of Indic languages is their com- plex morphology, which, when combined with the general lack of sufficient quanti- ties of high quality parallel data, can make developing machine translation (MT) for these languages difficult. In this paper, we describe our efforts towards develop- ing general domain English–Bangla MT systems which are deployable to the Web. We initially developed and deployed SMT- based systems, but over time migrated to NMT-based systems. Our initial SMT- based systems had reasonably good BLEU scores, however, using NMT systems, we have gained significant improvement over SMT baselines. This is achieved using a number of ideas to boost the data store and counter data sparsity: crowd transla- tion of intelligently selected monolingual data (throughput enhanced by an IME (In- put Method Editor) designed specifically for QWERTY keyboard entry for Devana- gari scripted languages), back-translation, different regularization techniques, dataset augmentation and early stopping. c⃝ 2018 The authors. This article is licensed under a Creative Commons 3.0 licence, no derivative works, attribution, CC- BY-ND. 1 Introduction Today, machine translation (MT) is largely domi- nated by neural (NMT) and statistical MT (SMT), with NMT, by far, becoming the most prevalent among the two (Bahdanau et al., 2014; Bojar et al., 2017). The performance of the corpus-based ap- proaches to MT primarily depends on the availabil- ity of corpora to train them, specifically sufficient quantities of parallel data in a given language pair. This problem is exacerbated by NMT, which gen- erally needs larger quantities of parallel data, and has stricter requirements as to the cleanliness of that data. Unfortunately, large amounts of readily available parallel resources exist only for a small number of languages, e.g., OPUS (Tiedemann and Nygaard, 2004) and Europarl (Koehn, 2005), with very few sources of Indic language data. While Indian languages are widely spoken (in terms of native speakers), most of these languages have very little or no parallel resources available to build a general domain MT system (Khan et al., 2017; Singh et al., 2017). In the absence of readily available parallel corpora, comparable resources are often used to extract good quality parallel data from the web (Irvine and Callison-Burch, 2013; Wołk et al., 2015). In this direction also, Indic languages have a very few comparable resources. A clear indication can be found by examining the number of Wikipedia pages available for Indic lan- guages. We found only 57k pages are available for Bangla (no Indic Language has more than 125k pages), while a large number of European lan- guages have more than 1 million pages. Further- more, …

Content truncated to 500 words