Measuring Spurious Correlation in Classification: “Clever Hans” in Translationese

Proceedings of Recent Advances in Natural Language Processing, pages 196–206 Varna, Sep 4–6, 2023 https://doi.org/10.26615/978-954-452-092-2_022 196 Measuring Spurious Correlation in Classification: ”Clever Hans” in Translationese Angana Borah1, Daria Pylypenko1, Cristina Espa˜na-Bonet2, Josef van Genabith1,2 1Saarland Informatics Campus, Saarland University, Germany 2DFKI GmbH, Germany anganaborah9@gmail.com, daria.pylypenko@uni-saarland.de, {cristinae,josef.van genabith}@dfki.de Abstract Recent work has shown evidence of ”Clever Hans” behavior in high-performance neural translationese classifiers, where BERT-based classifiers capitalize on spurious correlations, in particular topic information, between data and target classification labels, rather than genuine translationese signals. Translationese signals are subtle (especially for professional translation) and compete with many other sig- nals in the data such as genre, style, author, and, in particular, topic. This raises the general question of how much of the performance of a classifier is really due to spurious correlations in the data versus the signals actually targeted for by the classifier, especially for subtle target signals and in challenging (low resource) data settings. We focus on topic-based spurious cor- relation and approach the question from two directions: (i) where we have no knowledge about spurious topic information and its distri- bution in the data, (ii) where we have some indi- cation about the nature of spurious topic corre- lations. For (i) we develop a measure from first principles capturing alignment of unsupervised topics with target classification labels as an in- dication of spurious topic information in the data. We show that our measure is the same as purity in clustering and propose a ”topic floor” (as in a ”noise floor”) for classification. For (ii) we investigate masking of known spurious topic carriers in classification. Both (i) and (ii) contribute to quantifying and (ii) to mitigating spurious correlations. 1 Introduction The term translationese refers to systematic lin- guistic differences between originally authored texts and translated texts in the same language (Gellerstam, 1986). Important aspects of transla- tionese have been identified in the linguistic lit- erature (Toury, 1980; Baker et al., 1993; Teich, 2012; Volansky et al., 2013), including source lan- guage interference, over-adherence to target lan- guage, simplification, explicitation, and implicita- tion. Translationese may manifest itself at lexi- cal, syntactic, semantic, and discourse-related lev- els of linguistic description. While translationese signals are subtle (especially for professional hu- man translation), corpus-based linguistic methods (Baker et al., 1993) and machine learning based classification methods (Volansky et al., 2013; Ra- binovich and Wintner, 2015; Rubino et al., 2016; Pylypenko et al., 2021) are able to reliably dis- tinguish between original and translated texts in the same language, genre, and style. While basic research focuses on identifying and categorizing aspects of translationese, research has also shown that translationese clearly impacts practical cross- lingual tasks that involve translated data (Singh et al., 2019; Zhang and Toral, 2019; Clark et al., 2020; Artetxe et al., 2020). Finally, translationese is sometimes regarded as (one of) the final fron- tier(s) of high-resource machine translation (Fre- itag et al., 2020, 2019; Ni et al., 2022). In this paper, we focus on translationese classifi- cation (into original O and …

Content truncated to 500 words