Kernel-based Approach for Automatic Evaluation of Natural Language Generation Technologies: Application to Automatic Summarization

Proceedings of Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing (HLT/EMNLP), pages 145–152, Vancouver, October 2005. c⃝2005 Association for Computational Linguistics Kernel-based Approach for Automatic Evaluation of Natural Language Generation Technologies: Application to Automatic Summarization Tsutomu Hirao NTT Communication Science Labs. NTT Corp. hirao@cslab.kecl.ntt.co.jp Manabu Okumura Precision and Intelligence Labs. Tokyo Institute of Technology oku@pi.titech.ac.jp Hideki Isozaki NTT Communication Science Labs. NTT Corp. isozaki@cslab.kecl.ntt.co.jp Abstract In order to promote the study of auto- matic summarization and translation, we need an accurate automatic evaluation method that is close to human evalua- tion. In this paper, we present an eval- uation method that is based on convolu- tion kernels that measure the similarities between texts considering their substruc- tures. We conducted an experiment us- ing automatic summarization evaluation data developed for Text Summarization Challenge 3 (TSC-3). A comparison with conventional techniques shows that our method correlates more closely with hu- man evaluations and is more robust. 1 Introduction Automatic summarization, machine translation, and paraphrasing have attracted much attention recently. These tasks include text-to-text language genera- tion. Evaluation workshops are held in the U.S. and Japan, e.g., the Document Understanding Con- ference (DUC)1, NIST Machine Translation Evalu- ation2 as part of the TIDES project, the Text Sum- marization Challenge (TSC)3 of the NTCIR project, and the International Workshop on Spoken Lan- guage Translation (IWSLT)4. These evaluation workshops employ human eval- uations, which are essential in terms of achieving 1http://duc.nist.gov 2http://www.nist.gov/speech/tests/mt/ 3http://www.lr.titech.ac.jp/tsc 4http://www.slt.atr.co.jp/IWSLT2004 high quality evaluations results. However, human evaluations require a huge effort and the cost is con- siderable. Moreover, we cannot automatically eval- uate a new system even if we use the corpora built for these workshops, and we cannot conduct re- evaluation experiments. To cope with this situation, there is a particular need to establish a high quality automatic evalua- tion method. Once this is done, we can expect great progress to be made on natural language generation. In this paper, we propose a novel automatic evaluation method for natural language generation technologies. Our method is based on the Ex- tended String Subsequence Kernel (ESK) (Hirao et al., 2004b) which is a kind of convolution ker- nel (Collins and Duffy, 2001). ESK allows us to calculate the similarities between a pair of texts tak- ing account of word sequences, their word sense se- quences and their combinations. We conducted an experimental evaluation using automatic summarization evaluation data developed for TSC-3 (Hirao et al., 2004a). The results of the comparison with ROUGE-N (Lin and Hovy, 2003; Lin, 2004a; Lin, 2004b), ROUGE-S(U) (Lin, 2004b; Lin and Och, 2004) and ROUGE-L (Lin, 2004a; Lin, 2004b) show that our method correlates more closely with human evaluations and is more robust. 2 Related Work Automatic evaluation methods for automatic sum- marization and machine translation are grouped into two classes. One is the longest common subse- quence (LCS) based approach (Hori et al., 2003; Lin, 2004a; Lin, 2004b; Lin and Och, 2004). The other is the N-gram based approach (Papineni et al., …

Content truncated to 500 words