Big Data in Finance and Quantitative Finance

Big Data in Finance is the future of Finance, and Quantitative Finance is the pillar of the modeling phase. Investment managers, traditional financial services companies and Fintech companies that embrace and invest in Big Data in Finance will benefit from an integrated approach on data management, quantitative finance, computer science and financial expertise.Big data continues to transform the landscape of various industries, particularly financial services. Many financial institutions are adopting big data analytics in order to maintain a competitive edge. Through structure and unstructured data, complex algorithms can execute trades using a number of data sources. We will discuss in this post Big Data in Finance and Quantitative Finance. Big Data in Finance is the future of finance with quantitative finance as a pillar in the modeling side.

Big Data and Finance

The vast proliferation of data and increasing technological complexities continue to transform the way industries operate and compete. Over the last two years, 90 percent of the data in the world has been created as a result of the creation of 2.5 quintillion bytes of data on a daily basis. Commonly referred to as big data, this rapid growth and storage creates opportunities for collection, processing and analysis of structured and unstructured data.

Financial services, in particular, have widely adopted big data analytics to inform better investment decisions with consistent returns. In conjunction with big data, algorithmic trading uses vast historical data with complex mathematical models to maximize portfolio returns.

Quantitative Finance

Quantitative finance, also known as mathematical finance, is a field of applied mathematics, concerned with financial markets.
Financial markets have the same features as complex systems: feedback, non-stationarity (potentially), many interacting agents, adaptation, evolution, single realization and are open systems. When we go multivariate we can add the curse of dimensionality and interpretation as important issues as well.

We have several candidates for this behavior: stochastic models, chaos and determinism. Time series and dependence between variable can also be linear or non-linear.

We observe well known stylized facts like non-Gaussian distribution of returns, short-term autocorrelation of return, long-term autocorrelation of volatility, bubbles and bursts and stranger than strange attractors to name the most important ones.

With these ingredients we have important modeling challenges as model risk and estimation. Quantitative finance has been focusing on the modeling of theses processes: Modern Portfolio Theory, Black-Litterman, Entropy pooling, Black-Scholes and all the derivatives models, CAPM, Factor models are some of the models that have been more successful. These models are and will be the backbone of modeling in finance. Criticism of the discipline (often preceding the financial crisis of 2007–08 by several years) emphasizes the differences between the mathematical and physical sciences and finance, and the resultant caution to be applied by modelers, and by traders and risk managers using their models. Notable here are Emanuel Derman and Paul Wilmott, authors of the Financial Modelers’ Manifesto.

Big Data in Finance

Big Data in Finance is a broader answer to all these issues adding to the traditional quantitative finance models all the Artificial intelligence / machine learning models, integrating all the data processes / management and dealing with structured and unstructured data. Quantitative finance is the essential element in the financial markets modeling. You can see that in the graph.

Machine Learning

Machine learning is a set of models that enable machines to generate knowledge from experience. This means that computers are learning from data without being explicitly programmed. Mathematically speaking many of the models are non-linear regressions.

There are many different machine learning algorithms for classification and prediction. The algorithms can be either categorized by learning style (supervised learning, unsupervised learning, reinforcement learning), or by similarity in form or function. In supervised learning we have: parametric/non-parametric algorithms, support vector machines, kernels, neural networks, and unsupervised learning: clustering, dimensionality reduction, recommender systems and deep learning.

Quantitative Hedge Funds using Big Data in Finance

Large quantitative CTA’s like Winton Capital and MAN AHL, systematic CTA’s and quantitative stock pickers are using this big data in finance approach. In the high frequency space these aspects are even more important as obviously the low latency requires all the data gathering, processing, modeling is even more integrated and fine-tuned.

Big Data in Finance – Pros and Cons

On the pros side of Big Data in Finance we have:

  • Models are improving the ability to deal with unstructured data
  • Machine learning is especially suitable for non-linear data processes
  • Models can “learn” and “adapt”

On the cons side

  • Overfitting / Estimation are a big issue not only for artificial finance but also traditional quantitative finance. Non-linear processes are specially complicated to estimate.
  • Interpretation sometimes is hard
  • Low signal to noise ratio in general

Big Data in Finance Modeling Integrated Theory

In order to deal with these issues and trade-offs that are so frequent in finance and modeling in general we have developed a Big Data Modeling Integrated Theory : a new modeling framework that will help practitioners.  We will see in that in the upcoming posts.


Big Data in Finance is the future of Finance, and Quantitative Finance is the pillar of the modeling phase. Investment managers, traditional financial services companies and Fintech companies that embrace and invest in Big Data in Finance will benefit from an integrated approach on new sources of data, quantitative finance, computer science and financial expertise.

We will see all this in detail in our MSc in Business Analytics

Alumni, Faculty, Management, News |

Leave a comment

Email will not be published




You can use these tags and attributes HTMl: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>