2010 Failures in English-Czech Phrase-Based MT

Proceedings of the Joint 5th Workshop on Statistical Machine Translation and MetricsMATR, pages 60–66, Uppsala, Sweden, 15-16 July 2010. c⃝2010 Association for Computational Linguistics 2010 Failures in English-Czech Phrase-Based MT ∗ Ondˇrej Bojar and Kamil Kos Charles University in Prague, Institute of Formal and Applied Linguistics ( ´UFAL) Malostransk´e n´amˇest´ı 25, Praha 1, CZ-11800, Czech Republic bojar@ufal.mff.cuni.cz, kamilkos@email.cz Abstract The paper describes our experiments with English-Czech machine translation for WMT101 in 2010. Focusing primarily on the translation to Czech, our additions to the standard Moses phrase-based MT pipeline include two-step translation to overcome target-side data sparseness and optimization towards SemPOS, a metric better suited for evaluating Czech. Unfor- tunately, none of the approaches bring a significant improvement over our standard setup. 1 Introduction Czech is a flective language with very rich mor- phological system. Translation between Czech and English poses different challenges for each of the directions. When translating from Czech, the word order usually needs only minor changes (despite the is- sue of non-projectivity, a phenomenon occurring at 2% of words but in 23% of Czech sentences, see Hajiˇcov´a et al. (2004) and Holan (2003)). A much more severe issue is caused by the Czech vo- cabulary size. Fortunately, this can be to a certain extent mitigated by backing-off to Czech lemmas if the exact forms are not available. We are primarily interested in the harder task of translating to Czech and most of the paper deals with this direction. After a brief specification of data sets, pre-processing and evaluation method in this section, we provide details on the issue of Czech vocabulary size (Section 2). We de- scribe our current attempts at generating Czech ∗The work on this project was supported by the grants EuroMatrixPlus (FP7-ICT-2007-3-231720 of the EU and 7E09003 of the Czech Republic), GA ˇCR P406/10/P259, and MSM 0021620838. Thanks to David Kolovratn´ık for the help with manual evaluation. 1http://www.statmt.org/wmt10/ word forms in Section 3. Partly due to the large vocabulary size of Czech, BLEU score (Papineni et al., 2002) correlates rather poorly with human judgments. We summarize our efforts to use a bet- ter metric in the model optimization in Section 4. The final Section 5 lists the exact configurations of our English↔Czech primary submissions for WMT10, including the back-off to lemmas we use for Czech-to-English. 1.1 Data and Pre-Processing Pipeline Throughout the paper, we use CzEng 0.9 (Bojar and ˇZabokrtsk´y, 2009)2 as our main parallel cor- pus. Following CzEng authors’ request, we did not use sections 8* and 9* reserved for evaluation purposes. As the baseline training dataset (“Small” in the following) only the news domain of CzEng (126k parallel sentences) is used. For large-scale ex- periments (“Large” in the following) and our pri- mary WMT10 submissions, we use all CzEng do- mains except navajo and add the EMEA corpus (Tiedemann, 2009)3,4 of 7.5M parallel sententes. As our monolingual data we use by default only the target side of the parallel corpus. For experi- ments reported here, we also use the monolingual data provided …

Content truncated to 500 words