How we build a model that supplements stock market investors’ intuition with big data and AI

An AI model that enhances human intuition instead of replacing it

Alternative data and Machine Learning are gaining rapid traction in the investment industry. However AI and Machine learning still is not able to beat the market or most regular investors in a real life environment, even though the internet is full of articles predicting stock prices with 99% accuracy (overfitting is no joke). One can safely say there are domains where humans outperform AI and there are domains where it is the other way around. This also goes for investing. But why only use either human input or AI input when you can also combine them?

In this article Ill explain how we build a model that uses both human intuition and AI in analysing the stock market, as well as some examples on how it works in practice. This will be followed up by more elaborate examples in subsequent articles.

Where humans have the edge over AI— developing investment narratives

Humans are way better than AI in and telling stories and recognizing stories. A five year old can tell a coherent story whereas even the most sophisticated AI (eg GPT -3) still struggles with telling a random story, let alone recognizing one. Ask an investor what narrative will affect the stock price of Microsoft and he will instantly have some idea that it relates to tech, working from home (and therefore Covid developments) and cloud computing. AI does not have that intuition.

Where AI has the edge over humans — data and patterns

Let humans find the narrative and let AI find the pattern

  • Human input → Which narratives are suspected to drive price. Experienced investors generally have a good sense on what news or narratives drive price. We can use this information to feed in to our model (see also the real life example further on).
  • AI input → deduce narrative time series from our alternative dataset (Financial Tweets, Financial headlines, Financial subreddits). This is done by:
    - Using our customized algorithm to assign a narrative to each headline or Tweet
    - Determine for each relevant headline (i.e. that is related to the narrative) its sentiment, emotional value etc by using a Machine Learning based sentiment/emotion classifier.
  • AI input → Analyse which narrative exposures are most important in explaining future price. We feed (lagged) price and our narrative time series to our model. From this our model can be trained on what narrative series are most important in explaining future price movements.
  • AI input → Having established a model we continuously keep monitoring how narratives develop and how this would influence future price movements. We do this by setting up a dashboard with all relevant metrics.

We did this by leveraging our inhouse dataset of news websites, financial tweets and Reddit data. As with all things in live, it helps to describe this with an example;

A real life example

The narrative time series— From the ideas investors have we can let AI determine narrative time series from our alternative dataset of millions of tweets and headlines. We plotted a subset of our narratives against the stock market returns of TSMC stock (a large semiconductor company). Already you see there is some relation.

Example of a narrative time series vs stock returns

Determining the most important narratives — From all narratives given we can let our model analyse which narratives are most important for (future) returns.

Most important narratives for TSM returns

Here you see the top 10 most important narratives. For TSM this means that when consumer news gets more exposure that is an important indicator for future price. Almost as important as the sentiment on news on Taiwan (second row). This in turn is followed by negative exposure of consumer headlines.

Monitoring narratives and run predictions — All these metrics (and many more) are then continuously updated on our dashboard. In this way the investors intuition can be quantified and monitored. Our engine and AI model is able to build time series for any narrative so all of our dashboard are unique and tailormade to the need of the investor or client. It helps them leveraging all their years of experience and intuition by means of our dataset and AI analyses.

Narrative-Investing.IO custom dashboard

This in a nutshell explains how we tried to build a model that offers the best of both worlds, human intuition combined with the rigor and insight from AI. In a following piece we will dive in to some more use cases (spoiler, scenarios analyses, trend watching, theme investing). And in case you are interested in analysing your own intuition and narratives by means of a (free) customized example dashboard, go to and let us know, we are happy to help.

Hoping to add some creative ways of looking at quantitative finance