Evaluating Factual Consistency of Texts with Semantic Role Labeling

Proceedings of the The 12th Joint Conference on Lexical and Computational Semantics (*SEM 2023), pages 89–100 July 13-14, 2023 ©2023 Association for Computational Linguistics Evaluating Factual Consistency of Texts with Semantic Role Labeling Jing Fan∗and Dennis Aumiller∗ and Michael Gertz Institute of Computer Science, Heidelberg University j.fan@stud.uni-heidelberg.de {aumiller, gertz}@informatik.uni-heidelberg.de Abstract Automated evaluation of text generation sys- tems has recently seen increasing attention, particularly checking whether generated text stays truthful to input sources. Existing meth- ods frequently rely on an evaluation using task- specific language models, which in turn allows for little interpretability of generated scores. We introduce SRLScore, a reference-free eval- uation metric designed with text summariza- tion in mind. Our approach generates fact tu- ples constructed from Semantic Role Labels, applied to both input and summary texts. A final factuality score is computed by an ad- justable scoring mechanism, which allows for easy adaption of the method across domains. Correlation with human judgments on English issummarization datasets shows that SRLScore competitive with state-of-the-art meth- ods and exhibits stable generalization across datasets without requiring further training or hyperparameter tuning. We experiment with an optional co-reference resolution step, but find that the performance boost is mostly out- weighed by the additional compute required. Our metric is available online at: https:// github.com/heyjing/SRLScore 1 Introduction One of the remaining issues that prevents produc- tive deployments of neural text summarization sys- tems is the low correlation of system outputs with human preferences. Among those, factuality, i.e., the agreement of facts in the generated summaries with those present in the input text, is not part of the general training objectives of models, which frequently leads to hallucinated facts that are detri- mental to perceived system performance (ter Hoeve et al., 2020; Fabbri et al., 2021). Prior work has therefore introduced metrics for automated test- ing of factuality in generated text (Goodrich et al., 2019; Kryscinski et al., 2020; Yuan et al., 2021), ∗Both authors contributed equally to this work. which allows for a more nuanced verification of model capabilities. In particular, one of the first rel- evant works by Goodrich et al. (2019) introduces the idea of representing text as a series of "fact tuples", in their case as (subject, predicate, object) triplets. Their method exhibits some as- sumptions about the underlying data, which ham- pers correlation with human ratings. For example, subject or object may vary for the same sentence meaning expressed using different syntactic struc- tures, e.g., active and passive forms. Semantic Role Labeling (SRL), however, allows for a syntac- tically independent meaning representation. Our metric, SRLScore, improves factuality evaluation, building on fact tuples similar to Goodrich et al. It distinguishes itself in several ways from existing approaches, though: 1. To account for a more nuanced fact represen- tation, we employ SRL to produce abstract representations of sentences that are indepen- dent of their syntactic formulations. 2. Fact tuples in SRLScore are generated on the input text instead of gold summaries; as a con- sequence, our method is reference-free, and may be applied …

Content truncated to 500 words