How AI, alternative data and insights from psychology resulted in an innovative stock market indicator

Did you ever notice that you are more cautious in traffic after there has been a large accident in the news? Then you are aware that sometimes the human brain takes shortcuts that behavioural economists call heuristics. Additionally, you are probably aware that alternative (big) data is giving rise to new real-time insights. Finally, the fact that artificial intelligence, AI or machine learning is rapidly changing the world probably also hasn’t gone by unnoticed. In this paper I am going to describe how these three components (behavioural psychology, alternative data and AI) can be used to create an indicator for future stock market movements. This paper is divided in to the three components already mentioned a few times, a behavioural psychology part, an alternative big data part and the AI part. Or just scroll down to see the final result compared to the S&P 500.

The psychology part

The alternative data part

Therefore at we have a datasets with financial news articles from well know websites, financial tweets and other sources of financial news combined to one large dataset with unstructured alternative data. With this data we can see to what news investors are exposed to. And here is the cool thing, if we think about our behavioural flaws discussed in the first part, we can make a prediction as to where investors’ availability bias is headed to. Put differently, is we know if investors are reading more negative or positive news, we can try to predict if their investment decisions will be positive (i.e. long) or if they will be negative (i.e. short). Unfortunately going through all this data and seeing for every news article, tweet etc whether it’s positive or negative is quite undoable. But here is the good news, we can let a computer do that and that is where the AI part comes in.

The AI part

To summarize the first two steps, first we discussed how people are influenced by what they read, second we showed that with alternative data we obtain what they read. The next (and hardest) part is actually reading and classifying. Since we have about 1000 daily news articles we have to let a computer to this, which we do by means of AI/Machine learning. We can use a model that is trained to classify a string of text as either optimistic (scoring it with a max of +1) to pessimistic (scoring it with a minimum of -1). How we do this is quite technical and will be explained in a different post, but the end result is a dataset of more than a million articles classified from +1 (optimistic) to -1 (pessimistic). Now the final step, checking whether the theory we proposed in the first part holds any truth when checking the data.

Tying it all together

The dashed line shows the S&P500, the other one our sentiment indicator. The vertical axis shows sentiment (negative to positive) and the horizontal axis is time. You can clearly see how sentiment was already decreasing before the market collapsed under COVID fear. Interestingly you also sentiment improving before the market recovered. So at the least we can see from that this plot that there is a relation between our indicator and S&P stock prices. This shows how new technology and improvement in AI is giving us a new way for looking at the market.

We publish this indicator every day as part of a larger report which you can find on: Indicators — Alternative Analytics ( . Here we also have added some additional innovative metrics, along with outputs from neural networks (in the reports section, to be found here: Reports — Alternative Analytics ( In a next series will discuss how these are build and how they should be read.

Hoping to add some creative ways of looking at quantitative finance

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store