Benefits and Challenges of Big Data Projects

Big Data is a term used to describe the growing volume, speed, variety, variability and complexity of huge blocks of information that also grow exponentially over time. In marketing, a Big Data project pays off when it starts from a business question, builds a clean data foundation and delivers visible results quickly.

Picture the budget meeting at a consumer goods company with a presence in six countries. The chief marketing officer asks for a larger media allocation for the next quarter and the chief financial officer asks a reasonable question in return. Which channels, in which markets, are actually producing the margin? The marketing team has plenty of data. It has web analytics, social metrics, retail panel information, loyalty program records and the CRM. What it does not have is a single view in which these sources agree. Each analyst arrives with a different number, the discussion becomes a debate about whose spreadsheet is right and the budget is approved on instinct. Nobody in the room doubts that the answer exists somewhere in the data. The company simply cannot reach it.

Marketing departments face increasing demands to deliver measurable results, anticipate customer behavior and personalize campaigns with precision. Big Data has emerged as a powerful ally in meeting these demands, because it gives access to large volumes of structured and unstructured information that, when properly analyzed, can transform decision-making and campaign execution. Yet launching a Big Data project in marketing is not without difficulty. It requires clear objectives, the right technology stack, skilled talent and strong data governance. This article examines both the benefits and the obstacles, and it proposes a sequence that lets a marketing leader show results in months and not years.

What are the benefits and challenges of Big Data projects in marketing?

In short, Big Data is data so large and complex that traditional data management tools cannot store or process it efficiently. For marketing and sales departments, big data is a new paradigm that must be adopted to match the digital needs of customers. The term also refers to the analysis of descriptive, predictive and prescriptive information from large volumes of data, which supports evidence-based decision-making. Descriptive analysis explains what happened, predictive analysis estimates what is likely to happen and prescriptive analysis suggests what to do about it.

The strategic argument has been made for some time. In a well-known Harvard Business Review article, Andrew McAfee and Erik Brynjolfsson present big data as a management revolution more than a technology change, illustrating it with companies such as Sears Holdings and PASSUR Aerospace. The emphasis on management is worth noting, because it matches what practitioners observe. The technology is available to nearly everyone. What differs among companies is the willingness to let evidence challenge established habits.

By combining Big Data and advanced analytics with marketing and sales management strategies, organizations can have a substantial impact in several areas, which are considered in the sections that follow. They are presented as a connected story and not as a list, since progress in one tends to unlock the next.

How does Big Data improve customer knowledge and personalization?

The first contribution is knowledge of the customer. Big Data can reveal not only who customers are, but also where they are, what they want, how they prefer to be contacted and when. A retail company that integrates point-of-sale records with its loyalty program and its online behavior can see that a segment of customers buys in stores on weekends but browses on the website on weekday evenings, and it can adjust the timing and channel of each message accordingly. That is a small insight with immediate commercial value.

The second contribution follows from the first. By exploring large volumes of data and analyzing behavioral patterns, the company can identify the personalization preferences of its customers. This is possibly one of the most commercially desired use cases of a Big Data project, and the evidence on its value is consistent. McKinsey reports that personalization most often drives a 10 to 15 percent revenue lift, with company-specific results ranging from 5 to 25 percent. A bank that knows which customers are approaching a life event, such as a first home purchase, can present a mortgage conversation at the right moment instead of sending the same promotion to its entire base.

The same foundation supports campaigns that act in real time. With access to geolocation data, social networks and purchase traces, it is possible to build advanced campaign models that execute actions for customers as their context changes. A quick-service restaurant chain can trigger an offer when a loyalty member is within a few blocks of a store at lunchtime, and a telecom operator can offer a roaming package when a customer's device reaches a foreign airport. In both cases the company is responding to a signal and not broadcasting a message.

Can Big Data improve retention, loyalty and marketing efficiency?

Big Data can help a company discover what influences customer loyalty and what brings customers back again and again. That is more valuable than it sounds, because the drivers of loyalty are often different from what management assumes. An insurer may believe that price keeps policyholders, only to find through the analysis of claims and contact center records that the speed of the first claim response is the stronger predictor of renewal. Once the driver is known, the company can invest in it and measure the outcome, and it can intervene with at-risk customers before they leave.

The effect on efficiency is equally important for the executive committee. Through Big Data processes, a company can determine the optimal marketing expenditure across multiple channels and keep improving its marketing programs through continuous testing, measurement and analysis. A marketing department that can show which combination of channels produces the lowest cost per qualified lead, and which campaigns attract customers who stay, is in a different position from one that justifies its budget through activity metrics. It speaks the language of the chief financial officer.

The benefits do not stop at marketing and sales. Demand models can also automate production orders, inventories and operations, so the organization plans its resources better. A manufacturer that connects its commercial forecast to its supply chain avoids both stockouts and excess inventory, and a distributor that anticipates seasonal demand can negotiate better terms with suppliers. This is where marketing data becomes an enterprise asset, and the discipline of Big Data consulting becomes relevant to the whole business and not only to the marketing team.

Why do Big Data projects in marketing fail, and how do you avoid it?

The challenges associated with using Big Data effectively can be especially daunting for marketing departments. Most analytics systems are not aligned with the data, processes and business rules of the commercial area, which means that technically correct analyses often fail to answer the questions that sales and marketing leaders actually ask. A dashboard that reports page views by the hour may be impeccable and still irrelevant to a director who needs to know which accounts are likely to buy this quarter.

Poor data quality is the most frequent cause of disappointment. Gartner research estimates that poor data quality costs organizations an average of at least $12.9 million a year, a figure that includes wasted effort, flawed decisions and missed opportunities. Marketing feels this cost in duplicated contacts, inconsistent customer records and campaigns sent to the wrong people. ICX has written about the practical side of this problem in the perfect recipe for a clean HubSpot database.

Other causes are organizational. Projects start with technology and not with questions, ownership is divided between marketing and IT without a clear sponsor, and the first deliverable arrives after a year, by which time enthusiasm has faded. Talent is also a constraint. Analysts who understand both statistics and the commercial business are scarce, which argues for starting with models that the existing team can build and maintain. A final cause is cultural. If decision makers continue to rely on instinct, the best analytics produce reports that no one reads. Leadership must therefore ask for evidence in meetings, so that the data has a place at the table.

The recommendation that follows from these challenges is to adopt Big Data with the aim of short-term results that produce objective management outcomes. Three steps, taken in order, help a company get there.

What is the right sequence to start a Big Data project?

The first step is the Big Data architecture. At the beginning, the company must prepare its system architecture so that large volumes of data can be extracted and visualized. This is the first major achievement, and in pursuing it the company puts in order many of the bad practices common in organizations, such as non-centralized data, information of little or no quality and silos that were never integrated. This objective gives the first results, and it brings order and control to the commercial, technology and general management areas alike. The choice of tools should follow the architecture and not precede it, and the Technology Selection Toolkit for CX Leaders offers a structured way to compare options.

The second step is an information dashboard. Today many companies still perform business intelligence analysis only with the information of each department, instead of building dashboards that unify information from several areas, and the processes remain manual. A natural question follows. Why not deliver a dashboard with centralized information that shows the main indicators of a department in a single panel? It begins to give management visible results, and it builds the credibility that later projects will need. The practice of data analytics and business intelligence is designed to make that step fast.

The third step is the first analytical models. The team should take an inventory of the main indicators and define what type of analysis it needs, and can extract, for each of them. The advice is to begin with descriptive models, which are quick to implement and help managers obtain the first results that support decisions, and only then to develop predictive models. Starting in the opposite order risks exhausting the team and failing to show short-term results. Descriptive work also exposes data problems early, which makes the later predictive models more reliable.

Industry scenes where Big Data changes marketing decisions

The value of the sequence becomes clear in specific settings. Consider a grocery retailer with a loyalty program of two million members. In the first phase it centralizes transaction data and builds a dashboard showing sales by category, store and customer segment. In the second, descriptive analysis reveals that a third of its most profitable customers have reduced their visits for two consecutive months. In the third, a predictive model scores each customer for defection risk, and the marketing team launches a targeted offer with a clear test group. The outcome can be measured within weeks, and the investment is justified by the margin that the recovered customers generate.

In financial services, a card issuer can combine transaction categories, digital behavior and service interactions to identify customers who are about to concentrate their spending with a competitor. In hospitality, a hotel group can integrate booking data, review sentiment and website behavior to adapt offers by segment. In telecom, the combination of usage patterns and contact center records feeds retention programs that save customers before they ask to cancel. In every case the pattern is the same, a clean data foundation, a unified view for decision makers and models that move from description to prediction.

For companies that already hold large amounts of information in their CRM, the opportunity is often closer than expected. ICX has explored this in leveraging your CRM data in a big data strategy, which shows how customer records, interactions and transactions can anchor the first analytical models.

What this means for the C-level

For the chief executive, a Big Data project in marketing is a decision about how the company will compete on knowledge of its customers. The question is not whether to collect more data, since most organizations already have more than they use, but whether leadership will insist that decisions about budget, pricing, channels and product be made with evidence. That insistence, more than any platform, is what turns analytics into advantage.

For the chief marketing officer, the priority is to link data to outcomes the board understands. Customer acquisition cost, retention, lifetime value and the contribution of each channel to margin are the metrics that earn credibility, and a unified architecture is the way to produce them consistently. For the chief financial officer, the project is an investment with staged returns. The first phase brings order and control, the second brings visibility and the third brings predictive capability, and each stage can be evaluated on its own merits before the next is funded.

For the chief information officer and the chief digital officer, the lesson is to treat marketing data as part of the enterprise architecture and to avoid creating another silo. When marketing, sales, service and operations draw on the same governed data, the company can personalize, forecast and plan with consistency. The chief customer officer benefits too, because the same models that detect defection risk can guide service and loyalty actions.

How to start next Monday morning

A seasoned consultant would begin with a question and not with a tool. Ask the commercial team which three decisions it makes most often that would benefit from better evidence, such as where to allocate media spend, which customers to prioritize for retention or which products to promote by market. Choose one of them, and write down the indicators that would inform it and where each of those indicators currently lives.

Then inventory the sources, name an owner for the data and measure its quality honestly, including duplicates, missing fields and conflicting definitions. Build the simplest possible dashboard that brings the relevant indicators together in one place, and share it with the people who make the decision. Within a few weeks the team will learn which data is reliable and which is not, and the first descriptive analysis will likely suggest an action that can be tested immediately. Only after that experience is it sensible to commit to larger investments in infrastructure and predictive modeling.

Throughout, keep in mind that Big Data is not a one-time project. It is a continuous capability that evolves with the market and with consumer expectations, and the organizations that treat it as such are the ones that turn their marketing teams into drivers of business impact and innovation.

How ICX helps

ICX Consulting helps marketing and commercial leaders turn data into decisions. Our work in data-driven marketing consulting defines the questions that matter, the indicators that answer them and the governance that keeps the data trustworthy, while our predictive analytics for marketing practice develops the models that follow once the foundation is in place. We implement the architecture and dashboards on platforms such as Tableau and HubSpot, so that insight reaches the people who sell and serve. If you are comparing options for the technology that will support the project, begin with the Technology Selection Toolkit for CX Leaders.

Frequently asked questions

How is Big Data different from traditional business intelligence?

Traditional business intelligence analyzes structured information, usually from one department, and reports on what has happened. Big Data handles larger volumes, a wider variety of sources and information that arrives continuously, and it supports predictive and prescriptive analysis. In practice the two coexist, and many companies reach Big Data capability by extending the dashboards they already trust.

Does a mid-sized company need Big Data?

It needs the discipline more than the label. A mid-sized company with a CRM, a website and a transactional system already has a meaningful volume of customer data. The relevant step is to unify and govern it, and then to apply descriptive and predictive analysis to the decisions that matter most. Scale can be added gradually.

How long before a Big Data project shows results?

If the project begins with architecture and a unified dashboard, visible results can appear within a quarter. Predictive models usually follow in later phases, after data quality has been established. A project that promises predictions in the first weeks, without addressing the foundation, tends to disappoint.

What skills does the team need?

The core mix is a business owner who defines the questions, a data engineer who builds and maintains the pipelines, and an analyst who can translate results into commercial recommendations. Many companies start with this small group, supported by an external partner, and add specialized profiles such as data scientists as the models become more advanced.

If your marketing team is rich in data but poor in answers, the problem usually lies in how the information is organized and used. Request a diagnostic with an ICX consultant and we will help you define the first questions, the architecture to answer them and a plan that produces visible results quickly.