How AI, alternative data and insights from psychology resulted in an innovative stock market indicator

A short walkthrough on how the combination of three exciting fields; behavioural finance, alternative data and machine learning leads to new insights in to the stock market

The psychology part

The fact that you are more cautious when you just witnessed an accident is called availability bias in the school of behavioural finance. It describes the notion that people judge the probability of something happening more likely when an example more easily comes to mind. So for example you probably overestimate the probability of a shark attack when you have just read about in the news. You could argue that the same goes for economic principles. People are more likely to believe inflation is coming when the news is full of it. Or that a recession is coming when everyone is tweeting about it. Or that a stock is going up when its all over Reddit. Point is that news can drive the market, rather than just report or follow on it. In this respect also the recent work of Nobel Prize winner Robert Shiller is relevant. His book Narrative Economics describes how stories can propel economic events. But how can one assess what people are reading, or what news they are exposed to? This is where alternative data and AI play their part.

The alternative data part

Traditional finance data is usually nice to work with, it is clean, nicely structured and most data is readily available. Unfortunately that is also it’s drawback, since everyone is looking at it, it is hard to gain an edge. Alternative data however usually is instructed and messy, but it is unique. One of the largest alternative data sources is the internet. Terabytes of data are generated every minute and with this indicator we want to take advantage of that.

The AI part

AI is getting smarter and smarter almost daily it seems. And more importantly, big AI libraries are available to everyone with a computer running Python. Google (TensorFlow) and Facebook (PyTorch) are so kind to make their AI libraries open source, which puts some really powerful tools at the tips of your fingers. In this case we will let AI do two tasks:
- Read headlines (ML zero shot sentiment classification)
- Predict returns over a certain horizon
For this paper we will only discuss the first task, the next one is for another post.

Tying it all together

So to recap, academic research shows how news can influence investor sentiment, we have alternative (big) data that contains all this news, and we have a computer that can interpret that news faster than any human can. Lets see how this all looks like in an interactive plot.



Hoping to add some creative ways of looking at quantitative finance

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store