Spell Checking Techniques for Replacement of Unknown Words and Data Cleaning for Haitian Creole SMS Translation

Proceedings of the 6th Workshop on Statistical Machine Translation, pages 470–477, Edinburgh, Scotland, UK, July 30–31, 2011. c⃝2011 Association for Computational Linguistics Spell Checking Techniques for Replacement of Unknown Words and Data Cleaning for Haitian Creole SMS Translation Sara Stymne Department of Computer and Information Science Link¨oping University, Sweden sara.stymne@liu.se Abstract We report results on translation of SMS mes- sages from Haitian Creole to English. We show improvements by applying spell check- ing techniques to unknown words and creating a lattice with the best known spelling equiva- lents. We also used a small cleaned corpus to train a cleaning model that we applied to the noisy corpora. 1 Introduction In this paper we report results on the WMT 2011 featured shared task on translation of SMS messages from Haitian Creole into English, which featured a number of challenges. The in-domain data avail- able is small and noisy, with a lot of non-standard language. Furthermore, Haitian Creole is a low re- source language, for which there are few language technology tools and corpora available. Our main focus has been to make the best pos- sible use of the available training data through dif- ferent ways of cleaning the data, and by replacing unknown words in the test data by plausible spelling equivalents. We have also investigated effects of dif- ferent ways to combine the available data in transla- tion and language models. 2 Baseline system We performed all our experiments using a stan- dard phrase-based statistical machine translation (PBSMT) system, trained using the Moses toolkit (Koehn et al., 2007), with SRILM (Stolcke, 2002) and KenLM (Heafield, 2011) for language model- ing, and GIZA++ (Och and Ney, 2003) for word alignment. We also used a lexicalized reordering model (Koehn et al., 2005). We optimized each system separately using minimum error rate train- ing (Och, 2003). The development and devtest data were available in two versions, as raw, noisy data, and in a clean version, where the raw data had been cleaned by human post-editors. The different subcorpora had different tokeniza- tions and casing conventions. We normalized punc- tuation by applying a tokenizer that separated most punctuation marks into separate tokens, excluding apostrophes that were suspected to belong to con- tracted words or Haitian short forms, periods for ab- breviations, and periods in URLs. There were often many consecutive punctuation marks; these were re- placed by only the first of the punctuation marks. In the English translations of the SMS data there were often translator’s notes at the end of the transla- tions. These were removed when introduced by two standard formulations: Additional Notes or transla- tor’s note/interpretation. In addition the translation marker The SMS [ ...] were removed. Case information was inconsistent, especially for SMS data, and for this reason we lower-cased all Haitian source data. On the English target side we wanted to use true-cased data, since we wanted case distinctions in the translation output. We based the true-casing on Koehn and Haddow (2009), who changed the case of the first …

Content truncated to 500 words