Measuring Consistency in Text-based Financial Forecasting Models

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 13751–13765 July 9-14, 2023 ©2023 Association for Computational Linguistics Measuring Consistency in Text-based Financial Forecasting Models Linyi Yang1,2∗, Yingpeng Ma1,2∗, Yue Zhang1,2† 1Institute of Advanced Technology, Westlake Institute for Advanced Study 2School of Engineering, Westlake University yanglinyi,mayingpeng,yuezhang@westlake.edu.cn Abstract Financial forecasting has been an important and active area of machine learning research, as even the most modest advantage in predic- tive accuracy can be parlayed into significant financial gains. Recent advances in natural language processing (NLP) bring the oppor- tunity to leverage textual data, such as earn- ings reports of publicly traded companies, to predict the return rate for an asset. However, when dealing with such a sensitive task, the consistency of models – their invariance un- der meaning-preserving alternations in input – is a crucial property for building user trust. Despite this, current financial forecasting meth- ods do not consider consistency. To address this problem, we propose FinTrust, an evalu- ation tool that assesses logical consistency in financial text. Using FinTrust, we show that the consistency of state-of-the-art NLP mod- els for financial forecasting is poor. Our anal- ysis of the performance degradation caused by meaning-preserving alternations suggests that current text-based methods are not suit- able for robustly predicting market informa- tion. All resources are available at https: //github.com/yingpengma/FinTrust. 1 Introduction NLP techniques have been used in various financial forecasting tasks, including stock return prediction, volatility forecasting, portfolio management, and more (Ding et al., 2014, 2015; Qin and Yang, 2019; Xing et al., 2020; Du and Tanaka-Ishii, 2020; Yang et al., 2020a; Sawhney et al., 2020). Despite the increased performance of NLP models on financial applications, there has been pushback questioning their trustworthiness, and robustness (Chen et al., 2022; Li et al., 2022). Recently, the causal expla- nation has been viewed as one of the promising *Equal contribution. Yingpeng Ma did this work during his internship at Westlake University. †Correspondence to: zhangyue@westlake.edu.cn. Negation Consistency Replace a word with its antonym Symmetric Consistency Reverse the order of a sentence Additive Consistency Combine pairs of inputs that shares the same label Transitive Consistency Replace the company name within the same sector FinTrust🎯 Figure 1: Examples of four consistency transformations used in FinTrust �. directions for measuring the robustness and thus improving the transparency of models (Stolfo et al., 2022; Feder et al., 2022). Among them, consis- tency has been viewed as a crucial feature, reflect- ing the systematic ability to generalize in semanti- cally equivalent contexts and receiving increasing attention in tasks such as text classification and entailment (Jin et al., 2020; Jang et al., 2022). Previous text-based financial forecasting meth- ods have mostly considered stock movement pre- diction based on various sources of data, including financial news (Xu and Cohen, 2018; Zhang et al., 2018), analyst reports (Kogan et al., 2009; Rekab- saz et al., 2017), and earnings conference calls (Qin and Yang, 2019; Keith and Stent, 2019; Li et al., 2020; Chen et …

Content truncated to 500 words