Using Semantic Role Labeling to Improve Neural Machine Translation

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 305–312 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 305 Perceived Text Quality and Readability in Extractive and Abstractive Summaries Julius Monsen, Evelina Rennes Department of Computer and Information Science Link¨oping University, Link¨oping, Sweden {julius.monsen, evelina.rennes}@liu.se Abstract We present results from a study investigating how users perceive text quality and readability in extractive and abstractive summaries. We trained two summarisation models on Swedish news data and used these to produce summaries of articles. With the produced summaries, we conducted an online survey in which the extractive summaries were compared to the abstractive summaries in terms of fluency, adequacy and simplicity. We found statistically significant differences in perceived fluency and adequacy between abstractive and extractive summaries but no statistically significant difference in simplicity. Extractive summaries were preferred in most cases, possibly due to the types of errors the summaries tend to have. Keywords: extractive summarisation, abstractive summarisation, automatic text summarisation 1. Introduction Approaches to the task of automatic text summarisation can be divided into two main categories - extractive and abstractive (Hahn and Mani, 2000). In extractive sum- marisation, the most relevant sentences are extracted from the source document and concatenated into a sum- mary. In abstractive summarisation, novel sentences that capture the most important information from the source document are generated. Summarisation systems are often evaluated with content-based measures such as ROUGE and BLEU that calculate scores based on overlapping n-grams. These automatic measures are limited and reflect hu- man judgements poorly (Kry´sci´nski et al., 2019). Moreover, they fail to evaluate critical features such as factual correctness, relevance of content, fluency and coherence. Celikyilmaz et al. (2021) surveyed a range of evalua- tion methods for Natural Language Generation (NLG) in general. One category of methods subject to this survey was human-centric evaluation methods. Even though human evaluation is expensive to execute and the results are difficult to reproduce due to a lack of consistency in how human evaluations are run, it is deemed the most important form of evaluation for de- veloping NLG systems. After all, the ultimate goal of NLG and text summarisation is to produce text that is valuable to people. Human evaluation is also essential as it is considered the gold standard when developing automatic measures. There have been extensive efforts to investigate and de- velop new frameworks for evaluation of summaries. An example of such a framework is the FFCI frame- work proposed by Koto et al. (2021). Four key di- mensions were identified across which to evaluate sum- maries. These were faithfulness (degree of factual con- sistency with the source), focus (precision of summary content relative to the reference), coverage (recall of summary content relative to the reference), and inter- sentential coherence (document fluency between adja- cent sentences). Several traditional metrics, ROUGE included, were benchmarked through human evalua- tion for each dimension. The general finding was that ROUGE lacks fine-grained interpretability and that embedding-based measures correlate better …

Content truncated to 500 words