Cross-Lingual Transfer of Cognitive Processing Complexity

Findings of the Association for Computational Linguistics: EACL 2023, pages 655–669 May 2-6, 2023 ©2023 Association for Computational Linguistics Cross-Lingual Transfer of Cognitive Processing Complexity Charlotte Pouw ILLC, University of Amsterdam∗ c.m.pouw@uva.nl Nora Hollenstein University of Copenhagen nora.hollenstein@hum.ku.dk Lisa Beinborn Vrije Universiteit Amsterdam l.beinborn@vu.nl Abstract When humans read a text, their eye move- ments are influenced by the structural com- plexity of the input sentences. This cognitive phenomenon holds across languages and recent studies indicate that multilingual language mod- els utilize structural similarities between lan- guages to facilitate cross-lingual transfer. We use sentence-level eye-tracking patterns as a cognitive indicator for structural complexity and show that the multilingual model XLM- RoBERTa can successfully predict varied pat- terns for 13 typologically diverse languages, despite being fine-tuned only on English data. We quantify the sensitivity of the model to structural complexity and distinguish a range of complexity characteristics. Our results in- dicate that the model develops a meaningful bias towards sentence length but also integrates cross-lingual differences. We conduct a con- trol experiment with randomized word order and find that the model seems to additionally capture more complex structural information. 1 Introduction Approximately 7,000 languages are currently spo- ken in the world, exhibiting differences at almost every level of linguistic organization (Eberhard et al., 2022). Nonetheless, psycholinguistic the- ories are predominantly supported by evidence from a handful of Indo-European languages (Nor- cliffe et al., 2015). Only recently, researchers have started to explore cross-linguistic differences in the neural implementation of language, uncover- ing both striking similarities across languages and empirical differences that cannot be explained by a unitary account (Malik-Moraleda et al., 2022). In natural language processing, multilingual lan- guage models are optimized for tasks such as ma- chine translation or cross-lingual information re- trieval (Conneau et al., 2020) and follow a linguis- ∗This research was developed when the first author was affiliated to Vrije Universiteit Amsterdam. tically naïve training regime. They are trained on dozens of languages simultaneously and do not account for typological differences between lan- guages. Nevertheless, their cross-lingual transfer performance sets new records, even in zero-shot settings (Pires et al., 2019). The ability to transfer knowledge across languages has been attributed to the shared vocabulary that is used for all lan- guages (Wu and Dredze, 2019) because it enables the reuse of common morphological roots for lan- guages from the same family. However, recent studies indicate that vocabulary sharing is not a pre- requisite for cross-lingual transfer (Artetxe et al., 2020) and that structural commonalities between languages play a more prevalent role in models (Karthikeyan et al., 2020). Human sentence processing is sensitive to struc- tural complexity. Eye movement data recorded during reading provide insights into cognitive pro- cessing patterns with a temporal accuracy of mil- liseconds (Winke, 2013). Structural processing difficulty materializes as regressions towards the complex region and an increase of fixations on that region (Clifton and Staub, 2011). For example, sentences with an object-relative structure trigger more regressions than sentences with more com- mon subject-relative clauses (Gordon …

Content truncated to 500 words