Improving evaluation and optimization of MT systems against MEANT

Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 434–441, Lisboa, Portugal, 17-18 September 2015. c⃝2015 Association for Computational Linguistics. Improving evaluation and optimization of MT systems against MEANT Chi-kiu Lo∗ NRC-CNRC Multilingual Text Processing National Research Council Canada 1200 Montreal Road, Ottawa, Ontario K1A 0R6, Canada ChiKiu.Lo@nrc-cnrc.gc.ca Philipp C. Dowling∗ TUM Fakultät für Informatik Technische Universität München Boltzmannstraße 3, 85748 Garching bei München, Germany dowling@cs.tum.edu Dekai Wu HKUST Human Language Technology Center HK University of Science and Technology Clear Water Bay Kowloon, Hong Kong dekai@cs.ust.hk Abstract We show that, consistent with MEANT- tuned systems that translate into Chinese, MEANT-tuned MT systems that trans- late into English also outperforms BLEU- tuned systems across commonly used MT evaluation metrics, even in BLEU. The re- sult is achieved by significantly improv- ing MEANT’s sentence-level ranking cor- relation with human preferences through incorporating a more accurate distribu- tional semantic model for lexical similar- ity and a novel backoff algorithm for eval- uating MT output which automatic seman- tic parser fails to parse. The surprising result of MEANT-tuned systems having a higher BLEU score than BLEU-tuned sys- tems suggests that MEANT is a more accu- rate objective function guiding the devel- opment of MT systems towards producing more adequate translation. 1 Introduction Lo and Wu (2013) showed that MEANT-tuned system for translating into Chinese outperforms BLEU-tuned system across commonly used MT evaluation metrics, even in BLEU. However, such phenomena are not observed in MEANT-tuned system for translating into English. In this pa- per, for the first time, we present MT systems for translating into English, which is tuned to a improved version of MEANT, also outperforms BLEU-tuned system across commonly used MT evaluation metrics, even in BLEU. The improve- ments in MEANT include incorporating more ac- curate distributional semantic model for lexical similarity and a novel backoff algorithm for eval- uating MT output which the automatic semantic parser failed to parse. Empirical results show that ∗This work was completed at HKUST. the new version of MEANT is significantly im- proved in terms of sentence-level ranking correla- tion with human preferences. The accuracy of MEANT relies heavily on the accuracy of the model that determines the lexi- cal similarities of the semantic role fillers. How- ever, the discrete context vector model based on the raw co-occurrence counts used in the original proposal of MEANT does not work well in predict- ing the similarity of the lexicons used in the ref- erence and machine translations. Recent work by Baroni et al. (2014) shows that word embeddings trained by predict models outperforms the count based models in various lexical semantic tasks. Baroni et al. (2014) argues that predict models such as word2vec (Mikolov et al., 2013) outper- form count based models on a wide range of lexi- cal semantic tasks. It is also common knowledge that raw co-occurrence counts do not work very well and performance can be improved when trans- formed by reweighing the counts for context infor- mativeness and dimensionality reduction. In con- trast …

Content truncated to 500 words