Are ACT’s Scores Increasing with Better Translation Quality?

Proceedings of the Eighth Workshop on Statistical Machine Translation, pages 408–413, Sofia, Bulgaria, August 8-9, 2013 c⃝2013 Association for Computational Linguistics Are ACT’s scores increasing with better translation quality? Najeh Hajlaoui Idiap Research Institute Rue Marconi 19 CH-1920 Martigny Switzerland Najeh.Hajlaoui@idiap.ch Abstract This paper gives a detailed description of the ACT (Accuracy of Connective Trans- lation) metric, a reference-based metric that assesses only connective translations. ACT relies on automatic word-level align- ment (using GIZA++) between a source sentence and respectively the reference and candidate translations, along with other heuristics for comparing translations of discourse connectives. Using a dictio- nary of equivalents, the translations are scored automatically or, for more accu- racy, semi-automatically. The accuracy of the ACT metric was assessed by human judges on sample data for English/French, English/Arabic, English/Italian and En- glish/German translations; the ACT scores are within 2-5% of human scores. The actual version of ACT is available only for a limited language pairs. Conse- quently, we are participating only for the English/French and English/German lan- guage pairs. Our hypothesis is that ACT metric scores increase with better transla- tion quality in terms of human evaluation. 1 Introduction Discourse connectives should preserve their sense during translation, as they are often ambiguous and may convey more than one sense depending on the inter-sentential relation (causality, conces- sion, contrast or temporal). For instance, since in English can express temporal simultaneity, but also a causal sense. In this paper, we present results of different Ma- chine Translation systems for English-to-French and English-to-German pairs. More specifically, we measure the quality of machine translations of eight English discourse connectives: although, even though, meanwhile, since, though, while, however, and yet, adopting different approaches. This quality is measured using a dedicated met- ric named ACT (Accuracy of Connective Transla- tion), a reference-based metric that assesses only connective translations. The paper is organized as follows. In Section 2, we present the ACT metric and its error rate. In section 3, we compare the ACT metric to previous machine translation evaluation metrics. Finally, we present the results of the different English-to- German and English-to-French MT systems (Sec- tion 4). 2 ACT Metric We described the ACT metric in (Hajlaoui and Popescu-Belis, 2013) and (Hajlaoui and Popescu- Belis, 2012). Its main idea is to detect, for a given explicit source discourse connective, its transla- tion in a reference translation and in a candidate translation. ACT then compares and scores these translations. To identify the translations, ACT first uses a dictionary of possible translations of each discourse connective type, collected from training data and validated by humans. If a reference or a candidate translation contains more than one pos- sible translation of the source connective, align- ment information is used to detect the correct con- nective translation. If the alignment information is irrelevant (not equal to a connective), it then com- pares the word position (word index) of the source connective alignment with the index in the trans- lated sentence (candidate or reference) and the set of candidate connectives …

Content truncated to 500 words