Approximating a Deep-Syntactic Metric for MT Evaluation and Tuning

Proceedings of the 6th Workshop on Statistical Machine Translation, pages 92–98, Edinburgh, Scotland, UK, July 30–31, 2011. c⃝2011 Association for Computational Linguistics Approximating a Deep-Syntactic Metric for MT Evaluation and Tuning∗ Matouˇs Mach´aˇcek and Ondˇrej Bojar Charles University in Prague Faculty of Mathematics and Physics Institute of Formal and Applied Linguistics Malostransk´e n´amˇest´ı 25, Prague {bojar,machacek}@ufal.mff.cuni.cz Abstract SemPOS is an automatic metric of machine translation quality for Czech and English fo- cused on content words. It correlates well with human judgments but it is computation- ally costly and hard to adapt to other lan- guages because it relies on a deep-syntactic analysis of the system output and the refer- ence. To remedy this, we attempt at approxi- mating SemPOS using only tagger output and a few heuristics. At a little expense in corre- lation to human judgments, we can evaluate MT systems much faster. Additionally, we de- scribe our submission to the Tunable Metrics Task in WMT11. 1 Introduction SemPOS metric for machine translation quality was introduced by Kos and Bojar (2009). It is inspired by a set of metrics relying on various linguistic fea- tures on syntactic and semantic level introduced by Gim´enez and M´arquez (2007). One of their best performing metrics was Semantic role overlapping: the candidate and the reference translation are rep- resented as bags of words and their semantic roles. The similarity between the candidate and the refer- ence is calculated using a general similarity measure called Overlapping. The formal definition may be found in Section 4. ∗ This work has been supported by the grants EuroMa- trixPlus (FP7-ICT-2007-3-231720 of the EU and 7E09003 of the Czech Republic), P406/10/P259, P406/11/1499, and MSM 0021620838. Instead of semantic role labels (not available for Czech), Kos and Bojar (2009) use TectoMT frame- work (ˇZabokrtsk´y et al., 2008) to assign a seman- tic part of speech defined by Sgall et al. (1986). In addition they use t-lemmas (deep-syntactic lemmas) instead of surface word forms, which most impor- tantly means that the metric considers content words only. In the following, we will use “sempos” to de- note the semantic part of speech and “SemPOS” to denote the whole metric by Kos and Bojar (2009). SemPOS correlates well with human judgments on system level, see Section 2 for a brief summary of how the correlation is computed. The main draw- back of SemPOS is its computational cost because it requires full parsing up to the deep syntactic level to obtain t-lemmas and semposes. In Section 3 we propose four methods which approximate t-lemmas and semposes without the deep syntactic analysis. These methods require only part-of-speech tagging and therefore they are not only faster but also eas- ier to adapt for other languages, not requiring more advanced linguistic tools. Gim´enez and M´arquez (2007) and Bojar et al. (2010) used different formulas to calculate the final overlapping.1 In Section 4, we examine variations of the formula, adding one version of our own. By combining one of the approximation tech- niques with one of the …

Content truncated to 500 words