An Empirical Investigation of Bias in the Multimodal Analysis of Financial Earnings Calls

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3751–3757 June 6–11, 2021. ©2021 Association for Computational Linguistics 3751 An Empirical Investigation of Bias in the Multimodal Analysis of Financial Earnings Calls Ramit Sawhney IIIT Delhi ramits@iiitd.ac.in Arshiya Aggarwal Adobe India arsaggar@adobe.com Rajiv Ratn Shah IIIT Delhi rajivratn@iiitd.ac.in Abstract Volatility prediction is complex due to the stock market’s stochastic nature. Existing re- search focuses on the textual elements of finan- cial disclosures like earnings calls transcripts to forecast stock volatility and risk, but ig- nores the rich acoustic features in the com- pany executives’ speech. Recently, new mul- timodal approaches that leverage the verbal and vocal cues of speakers in financial disclo- sures significantly outperform previous state- of-the-art approaches demonstrating the ben- efits of multimodality and speech. However, the financial realm is still plagued with a se- vere underrepresentation of various communi- ties spanning diverse demographics, gender, and native speech. While multimodal mod- els are better risk forecasters, it is imperative to also investigate the potential bias that these models may learn from the speech signals of company executives. In this work, we present the first study to discover the gender bias in multimodal volatility prediction due to gender- sensitive audio features and fewer female ex- ecutives in earnings calls of one of the world’s biggest stock indexes, the S&P 500 index. We quantitatively analyze bias as error disparity and investigate the sources of this bias. Our re- sults suggest that multimodal neural financial models accentuate gender-based stereotypes.1 1 Introduction Earnings calls are publicly available, quarterly con- ference calls where CEOs discuss their company’s performance and future prospects with outside ana- lysts and investors (Qin and Yang, 2019; Sawhney et al., 2020b). They consist of two sections: a prepared delivery of performance statistics, anal- ysis and future expectations, and a spontaneous question-answer session to seek additional informa- tion not disclosed before (Keith and Stent, 2019). 1Code & Implementation: https://github.com/ midas-research/multimodal-bias-naacl Researchers have studied the Post Earnings An- nouncement Drift (PEAD) to observe that state- ments made by upper management affect the way information is digested and acted upon impact- ing short-term price movements (Ball and Brown, 1968; Bernard and Thomas, 1989; Yang et al., 2020). Audio features contextualize text and connotate speaker’s emotional and psychological state (Fish et al., 2017; Jiang and Pell, 2017; Burgoon et al., 2015; Bachorowski, 1999). Hence, when used with textual features, audio features significantly deter- mine the effect of earning calls on the stock market (Qin and Yang, 2019; Yang et al., 2020). Past re- search has shown that audio features such as speak- ers’ pitch, intensity, etc. vary greatly across gen- ders (Mendoza et al., 1996; Burris et al., 2014; Latinus and Taylor, 2012). Moreover, female exec- utives are highly underrepresented in these earnings calls(Agarwal, 2019; Investments, 2017). The vari- ation in audio features is amplified by deep learning models due to a dearth of female training examples and is manifested as a …

Content truncated to 500 words