Evaluating Factuality in Text Simplification

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 7331 - 7345 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Evaluating Factuality in Text Simplification Ashwin Devaraj1 William Sheffield2,4 Byron C. Wallace3 Junyi Jessy Li4 1 Computer Science, 2 Mathematics, 4 Linguistics, The University of Texas at Austin 3 Khoury College of Computer Sciences, Northeastern University ashwin.devaraj@utexas.edu, sheffieldw@utexas.edu b.wallace@northeastern.edu, jessy@utexas.edu Abstract Automated simplification models aim to make input texts more readable. Such methods have the potential to make complex information accessible to a wider audience, e.g., provid- ing access to recent medical literature which might otherwise be impenetrable for a lay reader. However, such models risk introducing errors into automatically simplified texts, for instance by inserting statements unsupported by the corresponding original text, or by omit- ting key information. Providing more readable but inaccurate versions of texts may in many cases be worse than providing no such access at all. The problem of factual accuracy (and the lack thereof) has received heightened atten- tion in the context of summarization models, but the factuality of automatically simplified texts has not been investigated. We introduce a taxonomy of errors that we use to analyze both references drawn from standard simplifi- cation datasets and state-of-the-art model out- puts. We find that errors often appear in both that are not captured by existing evaluation metrics, motivating a need for research into en- suring the factual accuracy of automated sim- plification models. 1 Introduction Simplification methods aim to make texts more readable without altering their meaning. This may permit information accessibility to a wide range of audiences, e.g., non-native speakers (Yano et al., 1994), children (De Belder and Moens, 2010), as well as individuals with aphasia (Carroll et al., 1998) and dyslexia (Rello et al., 2013). Simpli- fication may also help laypeople digest technical information that would otherwise be impenetra- ble (Damay et al., 2006; Devaraj et al., 2021). Recent work has made substantial progress by designing sequence-to-sequence neural models that “translate” complex sentences into simplified ver- sions (Xu et al., 2016; Alva-Manchego et al., 2020). (1) [Original] There was no difference in operating time or perioperative complication rates. [Model simplified] However, there was not enough evi- dence to determine if there was an important difference in operative time or complication rates when compared to conventional surgery. (2) [Original] All studies were associated with methodolog- ical limitations. [Model simplified] All studies were of poor quality and had limitations in the way they were conducted. (3) [Original] On June 24 1979 (the 750th anniversary of the village), Glinde received its town charter. [Model simplified] On June 24 1979, the 750th anniver- sary of the village was renamed. (4) [Original] Others agreed with the federal court; they started marrying people in the morning. [Model simplified] Others agreed with the federal court; they started trying in morning. (5) [Original] In 2014, Mary Barra became CEO of General Motors, making her the first female CEO of a major automobile company. [Model …

Content truncated to 500 words