Analyzing the Use of Character-Level Translation with Sparse and Noisy Datasets

Proceedings of Recent Advances in Natural Language Processing, pages 676–684, Hissar, Bulgaria, 7-13 September 2013. Analyzing the Use of Character-Level Translation with Sparse and Noisy Datasets J¨org Tiedemann Department of Linguistics and Philology Uppsala University Uppsala, Sweden jorg.tiedemann@lingfil.uu.se Preslav Nakov Qatar Computing Research Institute Qatar Foundation, P.O. box 5825 Doha, Qatar pnakov@qf.org.qa Abstract This paper provides an analysis of character-level machine translation mod- els used in pivot-based translation when applied to sparse and noisy datasets, such as crowdsourced movie subtitles. In our experiments, we find that such character- level models cut the number of untrans- lated words by over 40% and are espe- cially competitive (improvements of 2-3 BLEU points) in the case of limited train- ing data. We explore the impact of char- acter alignment, phrase table filtering, bi- text size and the choice of pivot language on translation quality. We further compare cascaded translation models to the use of synthetic training data via multiple pivots, and we find that the latter works signifi- cantly better. Finally, we demonstrate that neither word- nor character-BLEU corre- late perfectly with human judgments, due to BLEU’s sensitivity to length. 1 Introduction Statistical machine translation (SMT) systems, which dominate the field of machine translation today, are easy to build and offer competitive per- formance in terms of translation quality. Unfor- tunately, training such systems requires large par- allel corpora of sentences and their translations, called bitexts, which are not available for most language pairs and textual domains. As a result, building an SMT system to translate directly be- tween two languages is often not possible. A com- mon solution to this problem is to use an interme- diate, or pivot language to bridge the gap in train- ing such a system. A typical approach is a cascaded translation model using two independent steps of translating from the source to the pivot and then from the pivot to the target language. A special case is where the pivot is closely related to the source language, which makes it possible to train useful systems on much smaller bitexts using character- level translation models. This is the case we will consider below, translating Macedonian to English via related languages, primarily Bulgarian. Our main contribution is the further analysis of such a setup. We show that character-level models can cut the number of untranslated words almost by half since translation involves many transfor- mations at the sub-word level. We further explore the impact of character alignment, phrase table pruning, data size, and choice of pivot language on the effectiveness of character-level SMT models. We also study the use of character-level translation for the generation of synthetic training data, which significantly outperforms all cascaded translation setups. Finally, we present a manual evaluation showing that neither word- nor character-BLEU correlate perfectly with human judgments. The remainder of the paper is organized as fol- lows: Section 2 presents related work. Section 3 discusses technical details about using character- level SMT models. Section 4 describes the exper- iments, and Section 5 …

Content truncated to 500 words