Top 5 problems with big data - and how to solve them

All posts
January 31, 2020

big-data

Pioneers are finding all kinds of creative ways to use big data to their advantage. Insights gathered from big data can lead to solutions to stop credit card fraud, anticipate and intervene in hardware failures, reroute traffic to avoid congestion, guide consumer spending through real-time interactions and applications, and much more.

The benefits of big data are felt by businesses too. 61% of companies state that big data is driving revenue because it is able to deliver deep insights into customer behavior. For most businesses, this view of their existing data means gaining a 360-degree view of their customers.

A report by CapGemini agrees, stating:

“Digital customer experience is all about understanding the customer, and that means harnessing all sources – not just analyzing all contacts with the organization, but also linking to external sources such as social media and commercially available data. For the digital supply chain, it is about collecting and interpreting the data from connected devices.”

Many SMEs use CRMs, in collaboration with social networks and marketing platforms, to store and analyze customer data.

So, for many organizations, the biggest problem is figuring out how to get value from this data. Only 27% of the executives surveyed in the CapGemini report described their big data initiatives as successful.

This indicates that there is a huge gap between the theoretical knowledge of big data and actually putting this theory into practice.

So what's the problem?

Top 5 big data problems

1. Finding the signal in the noise

It’s difficult to get insights out of a huge lump of data. Maksim Tsvetovat, big data scientist at Intellectsoft and author of the book Social Network Analysis for Startups, said that in order to use big data properly, "There has to be a discernible signal in the noise that you can detect, and sometimes there just isn’t one.

"Once we’ve done our intelligence on the data, sometimes we have to come back and say we just didn’t measure this right or measured the wrong variables because there's nothing we can detect here."

So one of the biggest issues faced by businesses when handling big data is a classic needle-in-a-haystack problem. Tsvetovat went on to say that, in its raw form, big data looks like a hairball, and scientific approach to the data is necessary.

"You approach it carefully and behave like a scientist, which means if you fail at your hypothesis, you come up with a few other hypotheses, and maybe one of them turns out to be correct."

2. Data silos

Data silos are basically big data’s kryptonite. What they do is store all of that wonderful data you’ve captured in separate, disparate units, that have nothing to do with one another and therefore no insights can be gathered from this data because it simply isn't integrated.

Data silos are the reason you have to crunch numbers to produce a monthly sales report. They’re the reason that C-level decisions are made at a snail's pace. They’re the reason your sales and marketing teams simply don’t get along. They’re the reason that your customers are looking elsewhere to take their business because they don’t feel their needs are being met, and a smaller, more nimble company is offering something better.

And the best way to eliminate data silos? It's simple: integrate your data.

3. Inaccurate data

Not only are data silos ineffective on an operational level, they are also fertile breeding ground for the biggest data problem: inaccurate data.

According to a report from Experian Data Quality, 75% of businesses believe their customer contact information is incorrect. If you’ve got a database full of inaccurate customer data, you might as well have no data at all. The best way to combat inaccurate data? Eliminating data silos by integrating your data.

4. Technology moves too fast

Larger corporations are more likely to fall prey to data silos, for such reasons as they prefer to keep their databases on-premises, and because decision making about new technologies is often slow.

One example cited in the CapGemini report is that stalwarts like telcos and utilities “...are noticing high levels of disruption from new competitors moving in from other sectors. This issue was mentioned by over 35% of respondents in each of these industries, compared with an overall average of under 25%.”

In essence, traditional players are slower to adopt technological advances and are finding themselves faced with serious competition from smaller companies because of this.

Big data is also fast data. Paul Maritz, pivotal chief executive officer of the EMC Federation, wrote that,

“If you can obtain all the relevant data, analyze it quickly, surface actionable insights, and drive them back into operational systems, then you can affect events as they’re still unfolding. The ability to catch people or things ‘in the act’, and affect the outcome, can be extraordinarily important.”

The ability to make fast decisions and quickly act on insights gained on big data is an advantage SMEs have over large corporations.

5. Lack of skilled workers

CapGemini's report found that 37% of companies have trouble finding skilled data analysts to make use of their data. Their best bet is to form one common data analysis team for the company, either through re-skilling your current workers or recruiting new workers specialized in big data.

You need to find employees that not only understand data from a scientific perspective, but who also understand the business and its customers, and how their data findings apply directly to them.


Data integration is key

Data integration is absolutely essential for getting the full advantage out of your big data. Data integration addresses the need for eliminating data silos so you can obtain deeper insight from big data.

In the book Big Data Beyond the Hype, the authors found that “...we see too many people treat this topic as an afterthought — and that leads to security exposure, wasted resources, untrusted data and more. We actually think that you should scope your big data architecture with integration and governance in mind from the very start.”

Not only will this save the janitorial work that is inevitable when working with data silos and big data, it also helps to establish veracity. In other words, it will increase the trustworthiness of your data, which will underpin the authority of any insight you gain from analysing your data.

Problems with Big Data PieSync

The first step to integrating your data is to clean up your data. Big data consultant Ted Clark, from the data consultancy company Adventag, said:

"80% of the work data scientists do is cleaning up the data before they can even look at it. They’re data custodians rather than analysts. Anything you've done more than three times, you should automate - it might take longer the first time but the other times you will save time and focus on an analysis."

How to clean and maintain your data

1. Remove duplicates

If you’re using multiple channels to capture data, such as through your website, customer care centre and marketing leads, you’re running the risk of collecting duplicate information. There are tools to help you remove duplicate data - for instance, if you work with Google Contacts, you can merge your contacts.

2. Verify new data

Set company-wide standards on verifying all new captured data before it enters the central database. Put in checks to see if the customer isn’t already in the system, or that they’re not in the system under a different name or under their email address.

3. Update data

Keep your data updated. You can do this by using parsing tools, which scans all incoming emails and updates contact information as it comes to hand.

4. Implement consistent data entry

Ensure that all employees are aware of company-wide data entry standards. For instance, each customer record has to have first and last names.


A foolproof way of keeping your contact database up-to-date and consistent between apps is to make sure your data is synced and integrated. With PieSync you can sync all your contacts two-ways and in real time to take the hassle out of contact management. Get started with a free trial now.

About Vanessa Rombaut

Vanessa is a wordsmith extraordinaire. Originally from Australia, she has travelled the world and the seven seas to write scintillating content for you to enjoy.

She likes books, travel, vintage films and sushi (not necessarily in that order).