Issues and Perspectives from 10,000 Annotated Financial Social Media Data

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 6106–6110 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 6106 Issues and Perspectives from 10,000 Annotated Financial Social Media Data Chung-Chi Chen⋆ Hen-Hsen Huang†‡ Hsin-Hsi Chen⋆‡ ⋆ Department of Computer Science and Information Engineering, National Taiwan University, Taiwan †Department of Computer Science, National Chengchi University, Taiwan ‡MOST Joint Research Center for AI Technology and All Vista Healthcare, Taiwan Abstract In this paper, we investigate the annotation of financial social media data from several angles. We present Fin-SoMe, a dataset with 10,000 labeled financial tweets annotated by experts from both the front desk and the middle desk in a bank’s treasury. These annotated results reveal that (1) writer-labeled market sentiment may be a misleading label; (2) writer’s sentiment and market sentiment of an investor may be different; (3) most financial tweets provide unfounded analysis results; and (4) almost no investors write down the gain/loss results for their positions, which would otherwise greatly facilitate detailed evaluation of their performance. Based on these results, we address various open problems and suggest possible directions for future work on financial social media data. We also provide an experiment on the key snippet extraction task to compare the performance of using a general sentiment dictionary and using the domain-specific dictionary. The results echo our findings from the experts’ annotations. Keywords: financial social media, sentiment analysis, market sentiment 1. Introduction Much effort is being applied to natural language process- ing (NLP) technologies in various domains. In the NLP community, many researchers have begun to use machine learning on financial and economic data. Recently, work- shops and shared tasks such as ECONLP’19, FinNLP’19, FNP’19, DSMM’18, FiQA’18, and SemEval’17 Task 5 have been organized. This reflects the increasing interest of NLP researchers in financial and economic domains. In this paper, we focus on financial social media data and of- fer an in-depth analysis of the data that are useful for future work. Mining crowd opinions has been a crucial challenge given the prevalence of smartphones and social media. Indeed, decision-makers have begun to consult crowd opinions in their deliberations. Many topic-specific social media plat- forms have been developed. The social trading platform is a typical example, which provides a forum for investors to discuss their ideas, trading strategies, and analyses of finan- cial instruments. On StockTwits, a leading social trading platform, investors use cashtags to indicate financial instru- ments. For example, Apple Inc. and Microsoft Corporation are tagged as $AAPL and $MSFT, respectively. Further- more, investors can also annotate their tweets with a bullish or bearish label. Financial tweets with the writer-labeled annotations have been used as training and test data (Li and Shah, 2017). However, the reliability of writer-labeled data is sometimes poor (Huang et al., 2018). For instance, tweet (T1) is labeled as bullish but lacks enough content. (T2), as- signed a bullish label, is a description of the trading volume of $DPW. Note that both soaring and collapsing …

Content truncated to 500 words