What is Big Data and What is it For?

Big data is the term for very large volumes of structured and unstructured information, such as transactions, emails, sensor readings, audio and video, that traditional tools cannot handle well. It is for turning that information into better decisions. What matters is not how big the data is, but what a company does with it.

It is Monday morning and the executive committee of a regional retail chain is reviewing last week's results. Sales in the northern stores fell, but nobody agrees on why. Marketing believes the promotion missed its audience. Operations suspects stockouts in two best-selling categories. Finance sees margin eroding in the same stores and cannot say whether discounts or product mix is responsible. Every function arrives with a spreadsheet that is correct on its own terms, and the sheets contradict each other. Meanwhile the company holds years of till receipts, loyalty card histories, web sessions, call center recordings and supplier feeds that could settle the argument in an afternoon, if they could be read together.

That scene is the real definition of the big data problem for most executives. The exponential growth of information has changed how businesses make decisions and relate to customers, yet the data usually sits in separate systems, in different formats, owned by different departments. Big data refers to the vast volume of data generated by social media, sensors, transactions and devices that, when analyzed correctly, reveals the patterns, trends and insights that drive strategic action. The remainder of this article explains where the term comes from, what it is for, which kinds of analysis it supports, and what a leadership team should do to turn it into advantage across industries. Ironically, when we talk about big data what matters is not how big the data is but what companies do with it.

What is Big Data, and where does the idea come from?

The term big data describes large amounts of data that can be structured or unstructured. Structured data is the information found in most databases, displayed in rows and columns with titles, such as an order table or a customer master. Unstructured data includes emails, videos, audio files, documents and spreadsheets, which carry meaning that a conventional database cannot easily query. Most of what a modern company generates falls into the second category, which is why a business that analyzes only its structured tables is looking at a small part of its own customer.

To understand why volume alone is a poor guide, it helps to turn to Gartner analyst Doug Laney, who, in a 2001 MetaGroup research note, "3D Data Management: Controlling Data Volume, Velocity and Variety", introduced the concept of the 3Vs. Volume refers to the amount of data, variety to the number of data types and sources, and velocity to the speed at which data is generated and must be processed. According to the 3V model, the challenge of managing big data lies in the expansion of all three properties together and not just in volume. A retailer with modest volume but streaming data from stores, web and mobile can have a harder problem than a manufacturer with large but orderly historical files.

The model has since been extended. Veracity, meaning how trustworthy the data is, and variability, the widening range of values a large data set can take, were added by later authors, and value, which addresses the need to quantify what business data is actually worth, completes the picture. The infographic published by Diya Soubra, "The 3V Slams That Define Big Data" in Data Science Central, illustrates how the original three grew into a longer list. For a leadership team the useful lesson from this evolution is that the later additions are management questions and not technical ones. Whether data is reliable and whether it is worth the cost of using it are decisions for the business to make.

Why is Big Data important to businesses?

The purpose of big data is to create value from the ability to analyze very large volumes of information, to bring into view the information that sits "asleep" on company servers, and to find patterns that provide a competitive edge in decision making. In practice this means learning to ask the data good questions, so that it can explain what is happening, why it is happening, and what else may happen next. Those three questions correspond to three kinds of analysis, each more demanding than the last.

Evidence that this effort pays has been building for more than a decade. In a widely cited 2011 study, the McKinsey Global Institute estimated that a retailer making full use of big data could increase its operating margin by more than 60 percent, and that effective use of data in US healthcare could create more than 300 billion dollars in value a year (McKinsey Global Institute, Big data: The next frontier for innovation, competition, and productivity). Those are estimates of potential, produced in 2011, and they should be read as an indication of scale and not as a forecast for any one company. The same report warned that the constraint would be people. It projected a shortage of 140,000 to 190,000 workers with deep analytical skills in the United States by 2018, along with 1.5 million managers and analysts able to use big data to make decisions. A decade later the shortage has changed shape, but the observation remains useful, because the binding constraint on data value is usually management capability and not storage or software.

Harvard Business Review reached a complementary conclusion from the management side. Writing on the management revolution that big data brings, the authors describe the difference between big data and conventional analytics as a matter of volume, velocity and variety, and argue that the real challenges are managerial. Senior decision makers must learn to ask the right questions and to rely on evidence, organizations need people who can find patterns in very large data sets and turn them into business information, and IT functions must integrate the relevant internal and external sources (Harvard Business Review, Big Data: The Management Revolution). The technology is rarely the obstacle.

What kinds of analysis does Big Data make possible?

The three families of analytics are known as descriptive, predictive and prescriptive. They are best understood as a ladder. Most companies are on the first rung, a few have reached the second, and very few operate on the third. Moving up the ladder does not mean abandoning the lower rungs, because each one depends on the data discipline built below it.

Descriptive analytics

Descriptive analytics uses historical data to identify behaviors and describe how things are being done. It is the most widely used form of analytics in companies, and its objective is to describe the current situation well enough to make decisions with a high degree of success. It reads business indicators to show what has happened and what is happening. For the retail committee above, descriptive analytics would reconcile the three spreadsheets into a single view of sales, stock and margin by store and category, which is often enough to end the argument and focus the conversation on action.

Predictive analytics

Predictive analytics builds models that forecast what will happen. It helps a company understand how its environment, internal and external, may change. A typical use is to estimate how many sales the business will make next month from the behavior pattern of the current year. Its purpose is to make forecasts using advanced mathematical methods such as statistics or machine learning, including methods that predict missing data. Imagine a consumer bank that uses transaction history to identify which customers are likely to move their payroll to a competitor within ninety days. The value is not in the score itself but in the retention conversation it allows the relationship manager to have before the customer has decided to leave. The question that defines this kind of analysis is simple to state and difficult to answer: what will happen? Does your business know?

Prescriptive analytics

Prescriptive analytics uses simulation and optimization techniques to indicate which path the company should take. It goes beyond predicting what will happen and recommends what can be done to improve, providing recommendations that maximize business indicators. This is the most advanced type of analysis, since it prescribes a suitable response through business rules that execute processes when a given indicator changes. A hotel group, for instance, can combine demand forecasts with its own cost structure to adjust room rates, overbooking limits and promotional offers automatically within rules set by revenue management. The human role shifts from producing the number to governing the rules that produce it.

One caution belongs here, because it is where many promising programs lose credibility. Data that is abundant but unreliable produces confident mistakes. Duplicate customer records, inconsistent product codes and unclear ownership of each field mean that two analysts can run the same query and report different answers, which is precisely the situation the retail committee faced at the start of this article. Treating veracity as a management responsibility, with named owners, agreed definitions and a visible quality measure, is what allows the more ambitious forms of analysis to be trusted. ICX discusses the discipline behind this in its guide to data governance, and in its article on leveraging CRM data in a big data strategy, which shows how the customer records a company already owns can anchor a wider data effort.

What does Big Data mean for customer experience and eCommerce?

Understanding and applying big data has become essential for companies that want to decide better, run operations more efficiently and improve the customer experience. The ability to process large volumes of structured and unstructured data opens the door to discovering patterns, predicting behavior and reacting to market demand in something close to real time. That insight reaches everything from product development to personalized marketing.

For eCommerce platforms such as Adobe Commerce, formerly known as Magento, the integration of big data analytics is especially valuable. It lets an online business analyze customer interactions, follow purchasing trends and personalize offers on the basis of user behavior. A specialty retailer that sees which products customers view together but rarely buy together can change its merchandising and its recommendations within days. These insights improve operational efficiency and also lift engagement and loyalty, because the right message reaches the right customer at the right moment. When big data is paired with a capable eCommerce platform, the combination supports better decisions, higher profitability and more sustainable growth in an increasingly competitive marketplace.

The same logic applies to business-to-business companies, though the data looks different. An industrial distributor may hold years of quotes, orders, service tickets and delivery records. Read together, they can reveal which customers are growing, which are shrinking quietly, and which quotes are lost on price rather than on lead time. For readers who want to examine the commercial side of this question, ICX has discussed the benefits and challenges of big data projects in the marketing department.

What this means for the C-level

The first implication is that big data is a business agenda with a technology component, not the reverse. The chief executive should expect a short list of decisions the company wants to make better, whether pricing, assortment, credit approval, churn prevention or service routing, and should insist that every data initiative be tied to one of them. A project that begins with a data lake and no decision in mind tends to deliver an impressive repository and little else. The HBR emphasis on asking the right questions applies directly here.

The chief commercial and marketing officers can use data to attract and convert with more precision, because a unified view of the customer shows which segments respond and which are simply expensive to reach. The chief financial officer gains earlier sight of margin, discounting and cost to serve, and can test the return on a campaign before the next budget cycle. Leaders of service and customer experience can retain and build loyalty by detecting dissatisfaction in call transcripts and tickets before it appears in a survey or a cancellation. Operations leaders gain efficiency from demand forecasts that reduce both stockouts and excess inventory. For every one of these roles, trust in the data is the precondition. If the committee spends its time disputing whose number is right, the analytics have not yet begun.

How do you start on Monday morning?

A pragmatic first step is to pick one decision that matters and trace the data behind it. Ask who makes the decision, what they look at today, what they wish they could see, and where that information lives. The exercise usually exposes three things quickly, namely duplicated definitions, missing links between systems and a lack of ownership. Fix those for a single decision before attempting a company-wide platform.

Then organize the effort around the lessons that Harvard Business Review draws from practice. Identify the data that matters to the business and collect it creatively from varied sources. Build models only as complex as the goals require. Create simple tools that people across the organization can understand and use. And change capabilities and culture so that analytical results are acted on at every level, from executives to frontline staff (Harvard Business Review, Making Advanced Analytics Work for You). Teams that treat the last point as a communication task, explaining why the data matters, tend to see adoption where those that treat it as a technical rollout do not.

How ICX helps

ICX Consulting helps executive teams connect data to commercial decisions. Our big data consulting and data analytics and business intelligence work starts from the decisions that matter, and our customer analytics and insights practice turns the resulting view of the customer into retention, loyalty and revenue actions. Where technology is needed, we implement it on the platforms the client already uses or selects with us, including Tableau and CRM environments. Leadership teams who want a structured way to begin can use the ICX Digital Transformation Toolkit.

Frequently asked questions

How much data does a company need before it counts as Big Data?

There is no fixed threshold. The 3V model suggests the test is whether the combination of volume, variety and velocity exceeds what your current tools handle comfortably. A mid-sized company with streaming web, store and service data can face a big data problem well before it reaches enormous file sizes.

Is Big Data only for large corporations?

No. The decisions improved by data, such as pricing, retention and stock, exist in companies of every size. Cloud platforms have lowered the entry cost, so a mid-market firm can begin with one decision and one clean data source, then widen the scope as value is proven.

What is the difference between Big Data and business intelligence?

Business intelligence mainly reports on structured, historical data, which corresponds to descriptive analytics. Big data adds unstructured sources and higher speeds and supports predictive and prescriptive analysis. In practice the two work together, and many companies extend their existing reporting before adding advanced models.

Where do most Big Data initiatives go wrong?

They usually fail on management, not on technology. Common causes are no decision tied to the data, no owner for data quality, and no change in how people work once the insight exists. Both the McKinsey and HBR analyses cited above point to talent and decision culture as the limiting factors.

The companies that gain most from data are the ones that decide first what they want to know. If your leadership team wants to find where information is already waiting to improve a decision, talk to an ICX consultant about which question your data should answer first.