Fresh Data Blog
Fresh Data Archive
What’s So New About Big Data?
Date: June, 2015 --
There’s been a lot of discussion of late about Big Data and, like the term or not, many have come to question its exact meaning and eventually how it differs from the type of “database marketing” that’s been taking place within the catalog and other industries for years. So let’s delve into a then and now comparison and really illustrate how, why and where the evolution to Big Data across the board has shifted the old paradigm.
While it may be looked back upon as a simpler time, back in the days when there were small(er) collective pools of customer data, early direct marketing pioneers did pretty complex things with modest tools. Those collective pools were created by industry agreement and helped build out the catalog industry on a national scale.
The pools were most often used by catalogers and publishers, but also by some of the retail chains, like Sears. Each company contributed customer data of what today we would call “mere basic” information. Name, address, gender, item purchased category. Sometimes additional and more sophisticated information was provided – children in household, income, hobby, home owned or rented, pets, etc. What you received from the pool was in proportion to what you contributed.
Every company had its own “secret sauce” of prospective customer indicators built up over the years and their competitors would have loved to see the recipe, but the rules were strict: you could never learn from the data pool’s manager which company had contributed which information.
The near relatives of these systems, marketing lists, live on in a modified form at numerous co-op database and other list/data/lead retailers, and they continue to prove their worth when used responsibly.
While these data sources were, and remain, relatively simple, they are important tools to test one’s hunches about one’s customers and undertake useful experiments with segmentation.
This is very efficient when a “sophisticated” target selection involves a potentially limitless number of variables. The pool system, however, would give you only a half-dozen or so “selects”. Perhaps your records indicate homes with pets are more likely to buy, so your query would generate more such names. And, generally you received from the pool more “value” than you invested, because the leads would not be in your file, the cost of the prospects you received was fairly modest, being primarily the overhead and margin of a processing company, and you learned a bit more about what worked and on whom.
So, is Big Data better? Is it the heir to the commercial list business and the catalog pool described above? Is it the future of marketing? Just what is it?
An emerging consensus on a definition is: Big Data represents the Information assets characterized by such a High Volume, Velocity and Variety to require specific Technology and Analytical Methods for its transformation into Value.
Stated another way: “Big Data is a lot of data of very different kinds coming in at such volumes and at such a high speed that the only way it can be turned into something of value (value being defined by the user) is to manipulate and analyze it with special tools and techniques.
For the marketing world, the most common “value” we seek is how to convince someone to take some action - buy something, give something, call someone, join something…... (Admittedly, this is crude. Of course the road to the sale is often strewn with intermediate steps of “getting to yes”.)
The promise of “Big Data” would be to enable us to predict with greater accuracy which people will accept our offers. Part of that prediction product could conceivably be critical elements of when the target will act, at what price, and perhaps how often. Additional “Values” could include things like optimum media for messaging, time of day of messaging, and frequency of messaging.
And Big Data and processing and analytics will differ by subject! For example, the Big Data needed for a municipal public transportation system to schedule optimal deployment of personnel and equipment would be significantly different from the Big Data needed by the sales department of an airline. They are each looking for answers to different questions. In short, the “Values” each seeks will be different, and it is likely that the algorithms and other analytical tools they would employ would differ, even when dealing with the same population.
At bottom, what was done with the data in the old pools was a primitive “proto Big Data” exercise. The exercise of finding correlations of, for example, pet ownership and interest in, or proclivity to buy your product was a relatively simple example of what will occur as large scale and scope data acquisition becomes common-place, and algorithms are perfected.
The struggle will be to determine which bundles of data, using what methods of analysis, will generate recognizable and significant correlations which marketers can use to accomplish their missions. As with testing a new promotions piece, we’ll have to test new “bundles” of data over and over with new theories, looking for indicators of probable consumer behavior, which will then inform our campaigns, and perhaps even our target populations.
Sometimes those bundles will surprise us. One study looked at stock market search volume data provided by Google Trends to see if it was possible to predict stock market moves. They found that increases in search volume for financially relevant search terms tend to precede significant drops in the markets.
We suspect that there are more than a few “senior” traders on the exchange floors who know this by “instinct”. “Big Data”, once captured and analyzed, will undoubtedly be critical for marketing professionals. And some of the data found relevant might not be “personal” like age and pets and boat ownership; it might be how outdoor temperatures are trending, pollen count, the state of a State’s economy, what time elementary school is over.
The challenge will be to find what correlates to what. In fact, the next “pools” of data that may be shared might not be customer personal data but data on combinations of weather, economic well-being, and pollen counts, and that’s only in May.
Another major issue is the accuracy of the data being aggregated into marketing or other databases. While some source data elements may be response or transactionally driven, and therefore highly accurate, other elements, and this is especially true for international data where the sale of actual personal information is often illegal under local privacy laws, may be modeled or inferred, meaning they are based on extrapolations or an educated set of assumptions. As a rule a marketer should understand that the further away you are from the point of collection for a given data point, the less confidence you should have in its accuracy and/or currency. When the latter type of elements are utilized en mass as part of customer profiling or modeling initiatives, one must begin to wonder about the nature of the product of those initiatives, given that when models are stacked on top of models, the copy of a copy effect of producing ever more marginal returns may start to come into play.
One other final key factor that differentiates between then and now is simply the sheer number of sources producing large sets of incoming data (CRM, Marketing Automation, DMPs, eCommerce, ESP, DM, Call Center, Point of Sale, ad nets, etc.) and plethora of communication channels available to marketers within which they are able to take action to engage consumers based upon said information.
So the answer to the question “Is Big Data just New Segmentation” is that it incorporates both segmentation and imagination in looking for correlations, sometimes in weird areas quite outside the behavior of the customer. It recognizes that the customer’s decisions are impacted by countless factors, which we have to discover. So, Big Data is not just “New Segmentation”. It’s a New Computational Industry that represents the next evolutionary step forward for the database and data-driven marketing industry.
The time is quickly approaching that all of us in this industry will be mastering new analytical tools and looking to new kinds of data, in huge quantities from both online and offline sources, to find and interact with customers. As more and more data is gathered it’s more important than ever to eliminate data silos and utilize tools, such as Data Services’ MarketView DMP, that allow for a more complete, 360 degree view of your marketing universe. Be sure that Data Services, Inc. will stay at the cutting edge of the services you’ll need to employ in the new era of Big Data.