Big Data Analytics in Financial Services

The terms big data, data science, and data analytics are thrown around a lot in the business and financial world. Until recently, even the experts didn’t really know what they meant or how companies should invest in them.

Now, big data solutions have been tried and tested by institutions of all kinds – and a lot of the mist has been cleared. Let’s take a look at what big data analytics is, how it has been used in the financial world, and some of the current trends of industry leaders.


What is Big Data?
What is Big Data Analytics in Financial Services?
The Emergence of Big Data
Where Big Data Currently Fits With Other IT Trends
Which Financial Services Use Big Data Analytics?
How are they using it?
Problems With Having So Much Information
Legal Issues
Future Trends of Big Data in Financial Services

What is BIG Data?


Data has been around for centuries. Basically, anything that’s written down can be considered data. This data has traditionally been kept in books and libraries to be analyzed by different people. Then came computers, and the amount of data recorded started to skyrocket. Still, all the way up to the early 2000s this data was manageable with databases and other techniques.

Now, most of the world’s content isn’t created by institutions, it’s created by everyone. Social media activity, GPS locations, credit card purchases, shares, likes, Wikipedia pages, photos, and videos are all being generated at a blistering pace. This data is different from anything seen before in five main ways, called the “five Vs” of Big Data:

  1. Volume – the amount of data available now is just insane – 400 hours of video are uploaded to YouTube every minute!
  2. Variety – This information isn’t neatly organized into rows in a table, it comes in every format imaginable. Video, images, sales figures, social media followings…
  3. Veracity – When collecting data from many sources, accuracy can’t be guaranteed. This is difficult for those trying to extract meaning from a dataset
  4. Velocity – The rate data collection is increasing exponentially every year
  5. Value – This new form of data is valuable in a different way to traditional data collected by companies

The term ‘Big Data’ refers to this tidal wave of messy, unstructured information that’s uploaded to the internet every day.

 What is Big Data Analytics in Financial Services?


Banks and insurance companies used to basically guess which people would likely pay back a mortgage on time, or wouldn’t accidentally burn their house down. These days, they rely on this new ‘Big Data’ to help them make better guesses.


Big data analytics involves collecting and analyzing this new wave of information. It can be historical data stored in a database somewhere, or information being collected right now by a bank, shopping center, or insurance company.

Analyzing large amounts of data can happen in many different ways. Complex systems and algorithms are built especially for this purpose, and every financial institution uses them. The data is analyzed for patterns that can be used to predict future trends, calculate risks, or to determine prices.

The Emergence of Big Data

In the early 2000’s, data was considered by many to be a solved problem. Relational databases worked quite well for the databases of the time. However, as the world became more connected with the internet, the amount of data being collected by companies began to explode.

The term big data was coined by O’Reilly Media in 2005 to describe the large sets of data that were too big to manage with traditional methods. This data isn’t always structured in an easy way and is collected from many different sources at the same time. This type of data requires more creative and sophisticated approaches to fully understand.


Processing Large Datasets – Hadoop

These datasets are usually far too big to fit on a single computer – even a very powerful one. And, it takes so much processing to analyze that even a single supercomputer could take days to work through it all. The Hadoop was developed to solve these problems, and today it is one of the most important tools for big data analytics.

Hadoop allows data scientists to distribute extremely large files over many different computers. The data can then be broken down into smaller chunks and process by many computers at the same time. This was a big breakthrough for big data. Buying a few hundred off the shelf servers is much cheaper than buying a state-of-the-art supercomputer – meaning companies of all sizes can get in on the action. Check out our article on Hadoop for more info on how it works.

Where Big Data Currently Fits With Other IT Trends

The four big trends in IT right now are:

  1. Mobile
  2. Internet of Things
  3. Cloud Computing
  4. Big Data

Mobile devices are now more popular that desktops and laptops. Everything from our lightbulbs to our cars are now connected to the internet. And, all of the data exchanged between all of these is stored in ‘the cloud’ in huge data centers around the world.

It’s these three trends combined that have really created big data. 90% of the world’s data was created in the last two years.

Cloud computing technologies have made it possible and affordable to analyze all of this new information.


Which Financial Services Use Big Data Analytics?


The main financial services are:

  • Credit unions
  • Banks
  • Credit card companies
  • Insurance companies
  • Accountancy companies
  • Consumer finance companies
  • Stock brokerages

These days, all of them use data analytics, and most of them are completely dependent on it to function. Banks, credit card companies, and credit unions use historical information to determine the risk level of borrowers. They set interest rates based on these figures. Insurance companies use all sorts of data to determine the risk of just about anything and set premiums accordingly.

Let’s take a look at some examples of how financial companies are leveraging bit data right now.

How are they using it?

There are many ways financial companies are using data analytics to improve their products, reduce costs, and make customers happy.

1. Improve Customer Segmentation


Banks can use customer purchasing data to find out which products, mortgages, or credit cards different types of people are likely to buy. They use this information to segment customers into groups with similar financial situations and goals – and laser-target their sales and advertising.

That means customers get more recommendations for products they probably want, and banks spend less time trying to sell to people who probably aren’t going to buy.

2. Developing New Products

That idea can be taken even further and used to develop new products that will sell. Combining customer data with economic trends can help predict what financial products will be popular this year, or next. Banks and insurance companies can develop and market the right products at the right time.

Consumers get more products that suit their needs, and banks waste less money and earn more profit.

3. Detecting Fraudulent Activities

Modern cyber threats on big companies are often extremely sophisticated and can cause serious damage. Every IT system has vulnerabilities, and security breaches can often go unnoticed for months. The sheer scale of IT systems now means that manual monitoring for threats just isn’t feasible. Data analytics is the new tool security engineers are using to fight cyber crime.

For example, a bank can monitor customer spending patterns to detect anything abnormal in real-time. If a usually conservative spender suddenly starts taking out loans and going on shopping sprees, big red flags will go off. This is a powerful tool to stop fraudsters and thieves in their tracks.

This helps keep theft and damage to a minimum, which keeps costs and prices down for customers.

4. Lowering Investment Risks

All financial companies earn a profit by making good investments. This could be making loans to people who’ll pay them back, or by selling car insurance to someone who is a safe driver. This used to be the work of experts who would make an educated guess as to which investments would likely pay off.

Nowadays, financial companies analyze huge databases of information to find out which investments will likely pay off, and which ones won’t. And they do it with surprising accuracy.

Banks use this to calculate the risk of different homebuyers and set interest rates. Insurance companies do the same with home insurance premiums.

5. Sentiment Analysis

One way that social media is leveraged by stoke brokers is with sentiment analysis – or opinion mining. This means using clever natural language processing algorithms to figure out what people really think based on what they say. These techniques are used to analyze millions of tweets and facebook comments to guess at what is happening in the world right now and react instantly.

Many companies use this to build trading algorithms that can quickly bet against stock prices immediately after disaster hits. If you are the first to understand that a plane has crashed, you can quickly bet that Boeing shares will drop, and earn a huge profit. You only need to be milliseconds faster than your competition for this to work. However, this can go horribly wrong, as were about to find out.

Problems With Having So Much Information


Having huge amounts of information can be extremely useful to companies and its customers. But, things can go wrong. One interesting report found that Target identified a teenage girl as pregnant based on her shopping habits while using her loyalty card. Target used this information to send a mailer to her home recommending maternity clothes. The father complained about this ‘mistake’, but later found out that Target knew the truth before he did!

This story highlights that the data being collected by large companies is very personal, and it’s not always obvious how it’s going to be used.

An even bigger concern is when data is compromised. Data breaches are so common that we don’t even hear about most of them. Check out this visualization of the biggest data breaches in the last few years. There are literally hundreds, and with names like Apple, JP Morgan Chase, and the US Military on the list, there is a big reason for concern.

Algorithms Can Be Wrong


Back in 2013, many trading companies were using sentiment analysis to monitor new on Twitter. Unfortunately, many of them were taking tweets a little too seriously. After a hack to Associated Press’s Twitter account, a fake tweet went out reporting that the White House had been bombed and that Barack Obama had been injured.

The algorithms jumped on this news, and within seconds $130bn was wiped off the US stock market. The stocks made a recovery afterward, but it’s a scary reminder that relying on inaccurate information can have huge consequences.

Legal Issues



Regulating bodies in all countries have caught on to the fact that data is important. They realize that ensuring big data is used safely and responsibly is vitally important to the safety of citizens and the economy.

Since the financial crisis of 2007/2008, banks all over the world have seen an exponential increase in the amount and detail of data they need to report to authorities. This is to protect citizens and prevent another financial collapse caused by poorly understood financial products.

Third Party Data and Privacy

It makes sense that financial companies want to combine their own collections of information with data from other firms. This allows them to get a deeper understanding of customers and trends – and gain a competitive advantage.

For example, a life insurance company might want to look at your purchase history from your local supermarket’s loyalty program. This would help them learn more about your lifestyle and calculate a more accurate premium.

There are strict legal requirements when sharing information about people, especially personal information. Financial services companies need to comply with these rules and stay transparent about what they are doing with our data.

Future Trends of Big Data in Financial Services


Big data analytics has been around for over ten years now. The early adopters have learned from mistakes and shown how companies can actually profit from using big data analytics. Here are some of the trends for big data in the financial sector.

Moving Beyond the Hype

The hype is settling down now for big data. CIOs are not as willing to spend millions just on experimentation, they want to know how the returns are going to roll in.

Taking a Reality Check

One common mistake is overestimating what insights data can reveal. We saw earlier how overreliance on a single data source such as sentiment analysis can result in disaster. Executives and data scientists are now realizing that big data analytics isn’t a solution to everything, and a little bit of humility is called for.

Machine Learning

Artificial intelligence and machine learning are closely linked with data science. To effectively understand unstructured data, algorithms need to be smart and learn from the past. Many of the tech giants like Google, Microsoft, IBM, and Amazon are now offering machine learning as a service. That means everyone now has access to the type of machine learning that powers Google searches.

This is important, as now you don’t have to be a huge enterprise to have access to these technologies.

Keeping Data Secure

Regulators are getting better at understanding big data and enforcing that it is used responsibly. There will be big consequences for firms that don’t do their best to keep data safe. These will come from both governments in the form of fines and penalties, and customers in the form of trust. This is becoming even more difficult with the emergence of the internet of things.

The financial institutions that take this seriously will thrive.

Moving to the Cloud

The IT world is moving to the cloud, and financial companies are the ones leading the way. Cloud computing is a large factor that has driven big data applications. It allows for computing on a scale not possible before. The enormous data sets can be distributed among thousands of servers in a data center and worked on by thousands of machines at a time. Financial institutions are now beginning to trust the public cloud more than before.

Developers and Data Scientists

There is a growing skills shortage in data science. It’s becoming a competitive necessity for most companies to start using data science to function. But, as a relatively new discipline, there simply aren’t enough skilled workers in the field. Universities are racing to open up enough courses and programs to start filling this void, but it’s already huge.

Some experts estimate that 100,000-190,000 data science jobs will go unfilled in the US from 2011-2020. That’s massive, and financial institutions need to be able to deal with it. To fill this void, many companies are turning to machine learning to analyze their data in bulk.

Financial Startups and Challenger Banks

The blistering pace of technological advancement is disrupting almost every industry in the world, and financial services are no different. Startups and smaller companies have less historical weight to carry, and they can adopt technology faster and with greater precision.

Many financial startups are using big data analytics to offer unique services to customers that the big boys just can’t right now.

Zopa is a peer-to-peer lending platform. Instead of borrowing money from a bank or credit union, you can borrow from thousands of individuals who are ready to lend. This is in direct competition to the bank’s way of doing things.

Smaller, challenger banks like Atom and Metro Bank are also taking advantage of technology to put themselves in positions to compete. They are leveraging this new technology to win over customers.

Big Data is Here to Stay

A tidal wave of data is being generated at a faster pace every year – in every form imaginable. Within this data lies valuable knowledge about the way the world currently works, what people think, and what is going to happen next. Data analytics is the secret to finding it.

The future of financial services will be dominated by big data analytics. It’s no longer an experiment, the benefits are huge and they are being enjoyed by companies all over the world. Now, the tools are available for smaller companies to get in on the action.

What This Means For You

Big data is still developing, and the next few years are going to have successes and disasters. You can expect your financial services to get better and smarter quickly as the big companies have better tools and face more competition. However, in the race to be the best, some companies are going to make mistakes. This will come at a big cost to them and their customers, so make sure you’re careful about who has your data!


Sam Palmer

Sam Palmer

Web Developer and Tech Writer
Sam Palmer

Latest posts by Sam Palmer (see all)