Perceptual Models of Machine-Edited Text

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 3909–3920 August 1–6, 2021. ©2021 Association for Computational Linguistics 3909 Perceptual Models of Machine-Edited Text Elizabeth M. Merkhofer,1 Monica-Ann Mendoza,1 Rebecca Marvin2 and John C. Henderson1 1 MITRE 7525 Colshire Dr, McLean, VA 22102 2 Independent {emerkhofer,mamendoza,jhndrsn}@mitre.org,rebecca.c.marvin@gmail.com Abstract We introduce a novel dataset of human judg- ments of machine-edited text and initial mod- els of those perceptions. Six machine-editing methods ranging from character swapping to variational autoencoders are applied to collec- tions of English-language social media text and scientific abstracts. The edits are judged in context for detectability and the extent to which they preserve the meaning of the orig- inal. Automated measures of semantic similar- ity and fluency are evaluated individually and combined to produce composite models of hu- man perception. Both meaning preservation and detectability are predicted within 6% of the upper bound of human consensus labeling. 1 Introduction Machine-editing systems produce new versions of text using text as input. They contribute to tasks such as automatic summarization, simplification, natural language generation and generative adver- sarial NLP systems. These tasks have communica- tive goals, for example shorter, more accessible, or more appropriate text, and system developers are encouraged to improve their correlation with human performance on these tasks. While the mea- sured task performance of machine-editing systems continues to improve, one might consider how hu- mans perceive machine-edited text compared to human-produced text. One-off human evaluation of editing systems is expensive, incomparable, and must be constantly repeated. In this work we make first attempts at direct, general-purpose modeling of human perception of these texts and develop a model of human perception as it relates to two goals: to maximally maintain the meaning of an original and be minimally perceptible as machine output. Approved for Public Release; Distribution Unlim- ited. Public Release Case Number 21-0320. ©2021 The MITRE Corporation. ALL RIGHTS RESERVED. We present a dataset of human judgments about detectability and meaning preservation for machine-edited text.1 This dataset consists of 14,400 judgments about contextualized pairs of machine-edited sentences. The original texts are English-language and come from two domains: sci- entific papers and social media. The edits are cre- ated by six different algorithms using a variety of techniques. By comparing trivial editors to more subtle approaches under the same evaluation frame- work, we move toward generic models of percep- tion of edited text. Our analysis finds high interannotator agreement and examines human preference among the six ma- chine editors that generated the candidates. Exist- ing measures of similarity and fluency are evaluated as models of perception. We find that reference- informed models come close to human consensus of meaning preservation and detectability. How- ever, language models that don’t have access to the reference text have less success as generic models of detection. This dataset and analysis constitute a step toward modeling meaning preservation and detectability under a variety of machine-editing conditions representative of the state of the prac- tice. 2 Background Machine editing is a component of multiple …

Content truncated to 500 words