• by Yoav Naveh , Op-Ed Contributor, May 20, 2014

Big Data is one of the most trending buzzwords of the last couple of years. It is perhaps one of the most misunderstood and misused buzzwords as well. Is bigger necessarily better? Moreover, people are taking the "big" part too literally. Simply having billions of records on hand to sort through isn't what makes a company a Big Data company. 

It’s no coincidence that the growth of RTB from nonexistent to more than 40 billion daily impressions  followed a similar path as previous ad technologies. After all, it seems like big data and tons of impressions would go hand in hand. But I’m not really sure they do.

It’s only natural to ask yourself, "What is all this Big Data doing for me?"  Is Big Data changing the way we consume ads? Is it affecting the bottom line? Is it even worth sifting through? But most importantly, do you have the right tools to sift through it and find useful and actionable information that might affect my marketing or revenue goals? Perhaps this adage is most apt: You can't tell how much someone weighs with a yardstick. Do we have the tools we need for the job? Good question.

Big Data vs bigger piles of data

It is a tremendous challenge to analyze hundreds of millions of impressions bought daily, but not a new one. RightMedia reported that it began to analyze 10 billion impressions per day more than ive years ago. Today we're using computers to do the same thing faster, but not differently.  Is that even "Big Data?" Maybe, but I'd argue that it's not. If the techniques are the same and we’re just doing it at larger scale, that’s great, just not the innovation we’ve come to expect. What the online advertising industry really needs are new answers to the question, "How do I sell more stuff?" 

Adding incremental impression volume to media plans through manually selecting audience segments isn't the panacea advertisers are looking for. The added benefit is just too small.  However, using a small sample to infer information and correlations on a seemingly completely different sample can make a big difference in how media is bought.

Being able to scale up, prospect, and discover hundreds of thousands  of new customers based on completely new information would be exciting for everyone in the media business. Effectively harnessing Big Data means being able to infer based on a small number of impressions and apply that across 500 billion impressions efficiently.  Big Data should be very good at solving that problem because it is about inferring information to expand upon.

Harnessing Big Data

The real challenge is intelligently managing the amount of data that is available today for the ever-increasing billions of impressions. Companies like AppNexus, MediaMath, Turn, BlueKai and eXelate are making it easier and easier to layer first- and third-party data, exponentially adding correlations to be identified and analyzed. We're now incorporating offline data into our online datasets as well. Merging all of this data together is starting to become a bigger challenge than what would traditionally be considered the “Big Data.” What we're really after is "Big Micro Data" -- thought I'm aware that doesn’t sound as catchy as Big Data.

Audience segmenting is one of the easier approaches to look at, but it doesn't actually utilize the strength of Big Data. Segments are designed  for us  to easily relate to a group of impressions as something we can make a decision on, as people. Getting that on a huge impression set is a big step, but it is not a leap in getting a better insight for what makes each impression unique.

And it doesn't help in understanding its importance to a campaign’s performance. To do that, we need a more granular, high-definition resolution picture of what makes up an impression. This is built out of a lot of very small data points: background color, # of different fonts, # of ads, # of words, # of sites the user visits every day, time user spends on each site, and more.

A lot of these parameters may seem relatively insignificant on their own, but together they allow us to really understand what drives performance, and how impressions are tied to one another on every specific campaign or piece of creative.

Not only we can get a better understanding for each impression, we can also more easily transfer our knowledge of what works from one impression to another. We no longer need the exact segment to match. We only need some (not just one) of those smaller attributes to show correlation. Big (micro) Data is much more fit to do that.  Continually   adding more of these micro attributes to our learnings is the big promise of big data and the future of online advertising, and something that simple segmentation can't achieve. 

Dave Morgan, founder of Simulmedia, recently told Jeff Lanctot, former head of advertiser and publisher solutions at Microsoft, "There isn't a need for a different flavor of established solutions. There is a need for breakthrough software that truly makes marketing more effective and efficient." And he's right. What our industry needs are platforms that truly learn and infer. Not just ones that come up with the same answers faster. The devil is in the details, and that is what Big (micro) Data is really all about.