Reference and Document Aware Semantic Evaluation Methods for Korean Language Summarization

Proceedings of the 28th International Conference on Computational Linguistics, pages 5604–5616 Barcelona, Spain (Online), December 8-13, 2020 5604 Reference and Document Aware Semantic Evaluation Methods for Korean Language Summarization Dongyub Lee1, Myeongcheol Shin2, Taesun Whang3, Seungwoo Cho2, Byeongil Ko2, Daniel Lee2, Eunggyun Kim2, Jaechoon Jo4∗ 1Kakao Corp. jude.lee@kakaocorp.com 2Kakao Enterprise Corp. {index.sh,john.w,kobi.k,daniel.e,json.ng}@kakaoenterprise.com 3Korea University, South Korea {taesunwhang}@korea.ac.kr 4Hanshin University, South Korea {jaechoon}@hs.ac.kr Abstract Text summarization refers to the process that generates a shorter form of text from the source doc- ument preserving salient information. Many existing works for text summarization are generally evaluated by using recall-oriented understudy for gisting evaluation (ROUGE) scores. However, as ROUGE scores are computed based on n-gram overlap, they do not reflect semantic meaning correspondences between generated and reference summaries. Because Korean is an aggluti- native language that combines various morphemes into a word that express several meanings, ROUGE is not suitable for Korean summarization. In this paper, we propose evaluation metrics that reflect semantic meanings of a reference summary and the original document, Reference and Document Aware Semantic Score (RDASS). We then propose a method for improving the correlation of the metrics with human judgment. Evaluation results show that the correlation with human judgment is significantly higher for our evaluation metrics than for ROUGE scores. 1 Introduction The task of text summarization is to generate a reference summary that conveys all the salient information of an original document. There are two strategies for this type of summarization (i.e., extractive and abstractive summarization). With the extractive approach, the most noticeable key sentences are extracted from the source and compiled into a reference (Zhong et al., 2019; Wang et al., 2019; Xiao and Carenini, 2019). The second approach is abstractive, with which a paraphrased summary is generated from the source (Zhang et al., 2018; Guo et al., 2018; Wenbo et al., 2019). The generated summary may not contain the same words that appear in the source document. Therefore, measuring factual alignment between the generated summary and source document is important (Kryscinski et al., 2019). Most summarization models are evaluated using recall-oriented understudy for gisting evaluation (ROUGE) (Lin, 2004a), which measures n-gram overlaps between generated and reference sum- maries. ROUGE has proven to have a high correlation with manual evaluation methods, such as pyra- mid (Nenkova et al., 2007) and TAC AESOP (Owczarzak and Dang, 2011). However, Louis (2013) showed that the correlation significantly decreased when only one reference summary was provided. Additionally, considering the process by which a person manually summarizes a document, ROUGE is limited, because it does not reflect semantic meanings between generated and reference summaries. For example, when a person summarizes a document, they tend to use words that are implicit while not al- ways using the explicit words from the original document. As the ROUGE score is computed based on an n-gram overlap, the score can be low even if two words have the same semantic meaning. Table 1 shows an example of the ROUGE limitation when applied to a Korean summarization. This …

Content truncated to 500 words