Fourteen Light Tasks for comparing Analogical and Phrase-based Machine Translation

Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers, pages 444–454, Dublin, Ireland, August 23-29 2014. Fourteen Light Tasks for Comparing Analogical and Phrase-based Machine Translation Rafik Rhouma RALI / DIRO Universit´e de Montr´eal rafikrhouma@live.fr Philippe Langlais RALI / DIRO Universit´e de Montr´eal felipe@iro.umontreal.ca Abstract In this study we compare two machine translation devices on twelve machine translation medical- domain specific tasks, and two transliteration tasks, altogether involving twelve language pairs, including English-Chinese and English-Russian, which do not share the same scripts. We imple- mented an analogical device and compared its performance to the state-of-the-art phrase-based machine translation engine Moses. On most translation tasks, the analogical device outperforms the phrase-based one, and several combinations of both systems significantly outperform each system individually. For the sake of reproducibility, we share the datasets used in this study. 1 Introduction A proportional analogy is a relation between 4 objets, x, y, z and t, noted [x : y :: z : t], which reads x is to y as z is to t. A formal proportional analogy, hereafter analogy, is a proportional analogy which involves a relationship at the graphemic level, such as [atomkraftwerken : atomkriegen :: kraftwerks : kriegs] in German. Analogical learning is a holistic learning paradigm (sketched in Section 2) which relies on proportional analogies for generalizing a training set. Lepage and Denoual (2005b) pioneered the application of analogical learning to Machine Transla- tion (MT). Different variants of their system have been tested within the IWSLT evaluation campaigns (Lepage and Denoual, 2005a; Lepage and Lardilleux, 2008; Lepage et al., 2008; Lepage et al., 2009). Since then, a number of studies have been investigating analogical learning for performing more specific machine translation tasks. Langlais et al. (2009) applied it to translating medical terms, and Langlais and Patry (2007) investigated the more specific task of translating unknown words, a problem simultane- ously investigated in (Denoual, 2007). Recently, Langlais (2013) applied formal analogies to transliterate English proper names into Chinese. Those works suggest, at least on the tasks investigated, that analogical translation typically shows bet- ter precision than phrase-based Statistical MT (SMT), but at a much lower recall. Still, the analogical devices tested in these works vary from task to task, making it difficult to draw a clear picture of the strengths and weaknesses of analogy-based translation. In this study, we perform a systematic compari- son of an analogical and a phrase-based MT engine for the translation of fourteen different testbeds. We also improve the state-of-the-art of analogical learning by revisiting the aggregation step of the process. In particular, we observe that ranking analogical candidates according to random forests improves the performance of the analogical device, over training a classifier, as proposed for instance in (Langlais, 2013). On each task we tackle, we report improvements to the state-of-the-art in analogical learning. In the remainder of this paper, we describe the principle of analogical learning and sketch our analogi- cal device in Section 2. We describe our experimental protocol in …

Content truncated to 500 words