UniCoRN: Unified Cognitive Signal ReconstructioN bridging cognitive signals and human language

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 13277–13291 July 9-14, 2023 ©2023 Association for Computational Linguistics UniCoRN: Unified Cognitive Signal ReconstructioN bridging cognitive signals and human language Nuwa Xi, Sendong Zhao∗, Haochun Wang, Chi Liu, Bing Qin and Ting Liu Research Center for Social Computing and Information Retrieval, Harbin Institute of Technology, China {nwxi,sdzhao,hcwang,cliu,bqin,tliu}@ir.hit.edu.cn Abstract Decoding text stimuli from cognitive signals (e.g. fMRI) enhances our understanding of the human language system, paving the way for building versatile Brain-Computer Interface. However, existing studies largely focus on de- coding individual word-level fMRI volumes from a restricted vocabulary, which is far too idealized for real-world application. In this paper, we propose fMRI2text, the first open- vocabulary task aiming to bridge fMRI time series and human language. Furthermore, to ex- plore the potential of this new task, we present a baseline solution, UniCoRN: the Unified Cognitive Signal ReconstructioN for Brain De- coding. By reconstructing both individual time points and time series, UniCoRN establishes a robust encoder for cognitive signals (fMRI & EEG). Leveraging a pre-trained language model as decoder, UniCoRN proves its effi- cacy in decoding coherent text from fMRI se- ries across various split settings. Our model achieves a 34.77% BLEU score on fMRI2text, and a 37.04% BLEU when generalized to EEG- to-text decoding, thereby surpassing the former baseline. Experimental results indicate the fea- sibility of decoding consecutive fMRI volumes, and the effectiveness of decoding different cog- nitive signals using a unified structure. 1 Introduction Language serves as a window into the cognitive processes unfolding within our minds, communi- cating a vast amount of information through its syntax and semantics (Pagel, 2017). Advances in cognitive neuroscience have enabled us to di- rectly observe the cognitive processes that underlie language use through the analysis of non-invasive cognitive signals, such as functional Magnetic Res- onance Imaging (fMRI) and electroencephalogram (EEG). However, this also poses a challenge in understanding the relationship between these sig- nals and the external stimuli that give rise to them ∗Corresponding author within the mind. Deciphering cognitive signals into human language not only enhances our grasp of the linguistic system, but also facilitates the develop- ment of practical brain-computer interfaces (BCIs) by leveraging our comprehension of decoded sig- nals (Wolpaw, 2007; Mudgal et al., 2020). Although brain decoding has gained great suc- cess from word-level to sentence-level decoding on EEG (Panachakel and Ramakrishnan, 2021; Wang and Ji, 2022), relatively little research has been dedicated to directly generating text, particularly complete sentences, from fMRI volumes. This is largely attributed to the challenges posed by the relatively low temporal resolution of fMRI, which makes it challenging to acquire word-level fMRI frames within a sentence. In this study, we pro- pose fMRI2text, the first open-vocabulary task that decodes fMRI time series into the corresponding texts under naturalistic settings. Despite the early efforts in fMRI decoding (Mitchell et al., 2008; Palatucci et al., 2009; Wang et al., 2020; Zou et al., 2021), these methods are limited …

Content truncated to 500 words