Accuracy and robustness in measuring the lexical similarity of semantic role fillers for automatic semantic MT evaluation

Copyright 2012 by Anand Karthik Tumuluru, Chi-kiu Lo, and Dekai Wu 26th Pacific Asia Conference on Language,Information and Computation pages 574–581 Accuracy and robustness in measuring the lexical similarity of semantic role fillers for automatic semantic MT evaluation Anand Karthik TUMULURU, Chi-kiu LO and Dekai WU HKUST Human Language Technology Center Department of Computer Science and Engineering Hong Kong University of Science and Technology {jackielo,aktumuluru,dekai}@cs.ust.hk Abstract We present larger-scale evidence overturning pre- vious results, showing that among the many al- ternative phrasal lexical similarity measures based on word vectors, the Jaccard coefficient most in- creases the robustness of MEANT, the recently in- troduced, fully-automatic, state-of-the-art semantic MT evaluation metric. MEANT critically depends on phrasal lexical similarity scores in order to au- tomatically determine which semantic role fillers should be aligned between reference and machine translations. The robustness experiments were con- ducted across various data sets following NIST Met- ricsMaTr protocols, showing higher Kendall correla- tion with human adequacy judgments against BLEU, METEOR (with and without synsets), WER, PER, TER and CDER. The Jaccard coefficient is shown to be more discriminative and robust than cosine similarity, the Min/Max metric with mutual infor- mation, Jensen Shannon divergence, or the Dice’s coefficient. We also show that with Jaccard coeffi- cient as the phrasal lexical similarity metric, indi- vidual word token scores are best aggregated into phrasal segment similarity scores using the geomet- ric mean, rather than either the arithmetic mean or competitive linking style word alignments. Further- more, we show empirically that a context window size of 5 captures the optimal amount of information for training the word vectors. The combined results suggest a new formulation of MEANT with signifi- cantly improved robustness across data sets. 1 Introduction We present larger-scale evidence overturning previous re- sults, showing that the Jaccard coefficient among the al- ternative lexical similarity measure based on word vec- tors most increases the robustness of MEANT, even more than that of the Min/Max metric with mutual informa- tion metric, as used by Lo et al. (2012) in their formu- lation of MEANT that outperformed BLEU (Papineni et al., 2002), NIST (Doddington, 2002), METEOR (Baner- jee and Lavie, 2005), PER (Tillmann et al., 1997), CDER (Leusch et al., 2006), WER (Nießen et al., 2000), and TER (Snover et al., 2006). MEANT, the fully-automatic, state-of-the-art semantic MT evaluation metric as introduced by Lo et al. (2012) uses the Min/Max metric with mutual information on word vectors as the similarity measure to score phrasal similarity of the semantic role fillers which is the match- ing criterion to align semantic frames. In achieving the same, word vectors are trained on a window size of 5 and use arithmetic mean to aggregate token similarity scores into segment similarity scores. We explore the potential of alternate similarity metrics on word vectors such as the Jensen Shannon divergence, the Dice’s coefficient and Jaccard coefficient apart from cosine similarity and the Min/Max metric with mutual in- formation employed by Lo et al. (2012) in their work. We show …

Content truncated to 500 words