How Does the Experimental Setting Affect the Conclusions of Neural Encoding Models?

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 6397–6404 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 6397 How Does the Experimental Setting Affect the Conclusions of Neural Encoding Models? Xiaohan Zhang, 1,2 Shaonan Wang, 1,2 Chengqing Zong 1,2,3 1 National Laboratory of Pattern Recognition, Institute of Automation, CAS 2 School of Artificial Intelligence, University of Chinese Academy of Sciences 3 CAS Center for Excellence in Brain Science and Intelligence Technology {xiaohan.zhang, shaonan.wang, cqzong}@nlpr.ia.ac.cn Abstract Recent years have witnessed the tendency of neural encoding models on exploring brain language processing using naturalistic stimuli. Neural encoding models are data-driven methods that require an encoding model to investigate the mystery of brain mechanisms hidden in the data. As a data-driven method, the performance of encoding models is very sensitive to the experimental setting. However, it is unknown how the experimental setting further affects the conclusions of neural encoding models. This paper systematically investigated this problem and evaluated the influence of three experimental settings, i.e., the data size, the cross-validation training method, and the statistical testing method. Results demonstrate that inappropriate cross-validation training and small data size can substantially decrease the performance of encoding models, especially in the temporal lobe and the frontal lobe. And different null hypotheses in significance testing lead to highly different significant brain regions. Based on these results, we suggest a block-wise cross-validation training method and an adequate data size for increasing the performance of linear encoding models. We also propose two strict null hypotheses to control false positive discovery rates. Keywords: Neural encoding, fMRI, Naturalistic stimuli 1. Introduction In recent years, neural encoding models have become increasingly important tools in studying the neural ba- sis of language (Nunez-Elizalde et al., 2019). Neural encoding models are data-driven methods that explore the associative mapping between the linguistic stimuli and the evoked brain response in the data. In the fMRI literature, they have been used to investigate the corti- cal representations of a broad range of linguistic fea- tures (Mitchell et al., 2008; Wehbe et al., 2014; Huth et al., 2016; Wang et al., 2020; Sun et al., 2020; Jain et al., 2020; Zhang et al., 2020; Caucheteux et al., 2021). Many of these studies formalized the encoding models as linear regression models that trained with a cross-validation method to predict the brain responses to language stimuli. The conclusions reached in these studies highly depend on the prediction performance of encoding models. As the use the encoding models increases, it is critical to know whether and how vari- ous language-irrelevant factors affect prediction perfor- mance and conclusions of encoding models. One im- portant but often neglected kind of such factors is the experimental settings. As a data-driven method, the experimental settings would undoubtedly affect the performance of encod- ing models. However, it is unknown whether and how this will further affect the neural encoding conclusions. Existing work using neural encoding varied from the amount of neuroimaging data, the training method, and …

Content truncated to 500 words