Continuous Space Reordering Models for Phrase-based MT

Continuous Space Reordering Models for Phrase-based MT Nadir Durrani Fahim Dalvi Qatar Computing Research Institute – HBKU {ndurrani,faimaduddin}@qf.org.qa Abstract Bilingual sequence models improve phrase-based trans- lation and reordering by overcoming phrasal independence assumption and handling long range reordering. However, due to data sparsity, these models often fall back to very small context sizes. This problem has been previously ad- dressed by learning sequences over generalized represen- tations such as POS tags or word clusters. In this paper, we explore an alternative based on neural network models. More concretely we train neuralized versions of lexicalized reordering [1] and the operation sequence models [2] using feed-forward neural network. Our results show improve- ments of up to 0.6 and 0.5 BLEU points on top of the baseline German!English and English!German systems. We also observed improvements compared to the systems that used POS tags and word clusters to train these models. Because we modify the bilingual corpus to integrate reordering opera- tions, this allows us to also train a sequence-to-sequence neu- ral MT model having explicit reordering triggers. Our moti- vation was to directly enable reordering information in the encoder-decoder framework, which otherwise relies solely on the attention model to handle long range reorder- ing. We tried both coarser and fine-grained reordering op- erations. However, these experiments did not yield any im- provements over the baseline Neural MT systems. 1. Introduction Source-target bilingual sequence models have been used suc- cessfully as feature in phrase-based SMT [3, 2]. They are based on minimal translation units, and overcome in- dependence assumption by handling non-local dependencies across phrasal boundaries, thus providing better translation and reordering mechanism. Such models however suffer from data sparsity and fall back to very small context sizes during test time. This shortcoming is addressed by learning factored models [4, 5, 6], learned over POS and morpholog- ical tags or using word classes [7, 8, 9].1 An alternative way to address data sparsity and learn bet- ter generalizations is to use continuous representations. Neu- ral networks (NN) have shown success in Statistical Ma- chine translation with n-best re-ranking [12, 13] or directly as a feature [14, 15] used during decoding. More recently, 1as obtained during GIZA training [10] or using brown clusters [11] attention-based encoder-decoder Recurrant Neural Network (RNN) model [16], which trains a single large neural net- work, has emerged as the new state-of-the-art in MT. In this work, we neuralize two commonly used reorder- ing models namely lexicalized reordering [1] and the oper- ation sequence model (OSM) [2] and integrate them as fea- ture in phrase-based MT. We convert word-aligned bi-text into a sequence of operations through a deterministic algo- rithm (See Algorithm 1 in [17] ), the resulting vocabulary and number of model parameters can become very large. A model trained on such representation may suffer from data sparsity. To overcome this, we separate the streams of source and target sequences and concatenate them to simulate the jointness. A feed-forward neural network is then trained on such concatenated n-gram sequences. The …

Content truncated to 500 words