Evaluating Sentiment Analysis in the Context of Securities Trading

Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pages 2094–2103, Berlin, Germany, August 7-12, 2016. c⃝2016 Association for Computational Linguistics Evaluating Sentiment Analysis in the Context of Securities Trading Siavash Kazemian and Shunan Zhao and Gerald Penn Department of Computer Science University of Toronto {kazemian,szhao,gpenn}@cs.toronto.edu Abstract There are numerous studies suggesting that published news stories have an im- portant effect on the direction of the stock market, its volatility, the volume of trades, and the value of individual stocks men- tioned in the news. There is even some published research suggesting that auto- mated sentiment analysis of news docu- ments, quarterly reports, blogs and/or twit- ter data can be productively used as part of a trading strategy. This paper presents just such a family of trading strategies, and then uses this application to re-examine some of the tacit assumptions behind how sentiment analyzers are generally evalu- ated, in spite of the contexts of their appli- cation. This discrepancy comes at a cost. 1 Introduction The proliferation of opinion-rich text on the World Wide Web, which includes anything from product reviews to political blog posts, led to the growth of sentiment analysis as a research field more than a decade ago. The market need to quantify opinions expressed in social media and the blogosphere has provided a great opportunity for sentiment analy- sis technology to make an impact in many sectors, including the financial industry, in which interest in automatically detecting news sentiment in or- der to inform trading strategies extends back at least 10 years. In this case, sentiment takes on a slightly different meaning; positive sentiment is not the emotional and subjective use of laudatory language. Rather, a news article that contains pos- itive sentiment is optimistic about the future finan- cial prospects of a company. Zhang and Skiena (2010) experimented with news sentiment to inform simple market neutral trading algorithms, and produced an impressive maximum yearly return of around 30% — even more when using sentiment from blogs and twitter data. They did so, however, without an appropri- ate baseline, making it very difficult to appreciate the significance of this number. Using a very stan- dard, and in fact somewhat dated sentiment ana- lyzer, we are regularly able to garner annualized returns over twice that percentage, and in a man- ner that highlights two of the better design deci- sions that Zhang and Skiena (2010) made, viz., (1) their decision to trade based upon numerical SVM scores rather than upon discrete positive or nega- tive sentiment classes, and (2) their decision to go long (resp., short) in the n best- (worst-) ranking securities rather than to treat all positive (negative) securities equally. On the other hand, we trade based upon the raw SVM score itself, rather than its relative rank within a basket of other securities as Zhang and Skiena (2010) did, and we experimentally tune a threshold for that score that determines whether to go long, neutral or short. We sampled our stocks …

Content truncated to 500 words