Lancaster A at SemEval-2017 Task 5: Evaluation metrics matter: predicting sentiment from financial news headlines

Proceedings of the 11th International Workshop on Semantic Evaluations (SemEval-2017), pages 581–585, Vancouver, Canada, August 3 - 4, 2017. c⃝2017 Association for Computational Linguistics Lancaster A at SemEval-2017 Task 5: Evaluation metrics matter: predicting sentiment from financial news headlines Andrew Moore and Paul Rayson School of Computing and Communications, Lancaster University, Lancaster, UK initial.surname@lancaster.ac.uk Abstract This paper describes our participation in Task 5 track 2 of SemEval 2017 to pre- dict the sentiment of financial news head- lines for a specific company on a contin- uous scale between -1 and 1. We tack- led the problem using a number of ap- proaches, utilising a Support Vector Re- gression (SVR) and a Bidirectional Long Short-Term Memory (BLSTM). We found an improvement of 4-6% using the LSTM model over the SVR and came fourth in the track. We report a number of different evaluations using a finance specific word embedding model and reflect on the effects of using different evaluation metrics. 1 Introduction The objective of Task 5 Track 2 of SemEval (2017) was to predict the sentiment of news headlines with respect to companies mentioned within the headlines. This task can be seen as a finance- specific aspect-based sentiment task (Nasukawa and Yi, 2003). The main motivations of this task is to find specific features and learning algorithms that will perform better for this domain as as- pect based sentiment analysis tasks have been con- ducted before at SemEval (Pontiki et al., 2014). Domain specific terminology is expected to play a key part in this task, as reporters, investors and analysts in the financial domain will use a specific set of terminology when discussing fi- nancial performance. Potentially, this may also vary across different financial domains and indus- try sectors. Therefore, we took an exploratory ap- proach and investigated how various features and learning algorithms perform differently, specifi- cally SVR and BLSTMs. We found that BLSTMs outperform an SVR without having any knowl- edge of the company that the sentiment is with re- spect to. For replicability purposes, with this paper we are releasing our source code1 and the finance specific BLSTM word embedding model2. 2 Related Work There is a growing amount of research being car- ried out related to sentiment analysis within the financial domain. This work ranges from domain- specific lexicons (Loughran and McDonald, 2011) and lexicon creation (Moore et al., 2016) to stock market prediction models (Peng and Jiang, 2016; Kazemian et al., 2016). Peng and Jiang (2016) used a multi layer neural network to predict the stock market and found that incorporating textual features from financial news can improve the accu- racy of prediction. Kazemian et al. (2016) showed the importance of tuning sentiment analysis to the task of stock market prediction. However, much of the previous work was based on numerical finan- cial stock market data rather than on aspect level financial textual data. In aspect based sentiment analysis, there have been many different techniques used to predict the polarity of an aspect as shown …

Content truncated to 500 words