DCU-Lingo24 Participation in WMT 2014 Hindi-English Translation task

Proceedings of the Ninth Workshop on Statistical Machine Translation, pages 215–220, Baltimore, Maryland USA, June 26–27, 2014. c⃝2014 Association for Computational Linguistics DCU-Lingo24 Participation in WMT 2014 Hindi-English Translation task Xiaofeng Wu, Rejwanul Haque*, Tsuyoshi Okita Piyush Arora, Andy Way, Qun Liu CNGL, Centre for Global Intelligent Content School of Computing, Dublin City University Dublin 9, Ireland {xf.wu,tokita,parora,away,qliu}@computing.dcu.ie *Lingo24, Edinburgh, UK rejwanul.haque@lingo24.com Abstract This paper describes the DCU-Lingo24 submission to WMT 2014 for the Hindi- English translation task. We exploit miscellaneous methods in our system, including: Context-Informed PB-SMT, OOV Word Conversion (OWC), Multi- Alignment Combination (MAC), Oper- ation Sequence Model (OSM), Stem- ming Align and Normal Phrase Extraction (SANPE), and Language Model Interpola- tion (LMI). We also describe various pre- processing steps we tried for Hindi in this task. 1 Introduction This paper describes the DCU-Lingo24 submis- sion to WMT 2014 for the Hindi-English transla- tion task. All our experiments on WMT 2014 are built upon the Moses phrase-based model (PB-SMT) (Koehn et al., 2007) and tuned with MERT (Och, 2003). Starting from this baseline system, we exploit various methods including Context- Informed PB-SMT (CIPBSMT), zero-shot learn- ing (Palatucci et al., 2009) using neural network- based language modelling (Bengio et al., 2000; Mikolov et al., 2013) for OOV word conversion, various lexical reordering models (Axelrod et al., 2005; Galley and Manning, 2008), various Mul- tiple Alignment Combination (MAC) (Tu et al., 2012), Operation Sequence Model (OSM) (Dur- rani et al., 2011) and Language Model Interpola- tion(LMI). In the next section, the preprocessing steps are explained. In Section 3 a detailed explanation of the technique we exploit is provided. Then in Sec- tion 4, we provide our experimental results and re- sultant discussion. 2 Pre-processing Steps We use all the training data provided for Hindi– English translation. Following Bojar et al. (2010), we apply a number of normalisation methods on the Hindi corpus. The HindEnCorp parallel cor- pus compiles several sources of parallel data. We observe that the source-side (Hindi) of the TIDES data source contains font-related noise, i.e. many Hindi sentences are a mixture of two different en- codings: UTF-81 and WX2 notations. We pre- pared a WX-to-UTF-8 font conversion script for Hindi which converts all WX encoded characters into UTF-8, thus removing all WX encoding ap- pearing in the TIDES data. We also observe that a portion of the English training corpus contained the following bracket- like sequences of characters: -LRB-, -LSB-, - LCB-, -RRB-, -RSB-, and -RCB-.3 For consis- tency, those character sequences in the training data were replaced by the corresponding brackets. For English – both monolingual and the target side of the bilingual data – we perform tokeniza- tion, normalization of punctuation, and truecasing. For parallel training data, we filter sentences pairs containing more than 80 tokens on either side and 1http://en.wikipedia.org/wiki/UTF-8 2http://en.wikipedia.org/wiki/WX_notation 3The acronyms stand for (Left|Right) (Round|Square|Curly) Bracket. 215 sentence pairs with length difference larger than 3 times. 3 Techniques Deployed 3.1 Combination of Various Lexical Reordering Model (LRM) Clearly, Hindi and English have …

Content truncated to 500 words