Commentary
The Numbers Just Don't Add Up
- by Jim Sterne , October 2, 2009
Between the realization of an eye-opening, game-changing insight gleaned from advertising test results and Web behavior data, the report you're gleefully ferrying to the C-Suite wilted, turns brown at the edges and starts to dribble a slimy substance with a conspicuous stench.
The CMO immediately develops a nose-squint. The VP of Corporate Communications has that "Oooo, you're in for it!" look in her eye and the VP of Advertising nudges the Director of Direct Marketing and says sotto-voce, "The golden boy is about to find out his day in the sun has turned him to toast."
The CMO points to (but does not touch)
a traffic report from comScore
a traffic report from Hitwise
a chart from Compete.com
an ad banner report from Atlas
a traffic report from Omniture and
another from Google Analytics
"It's like the old joke," she said with no humor at all. "If you take all the economists in the world and line them up end-to-end, they all point different directions. What the hell is going on with these numbers? Are we getting thirty two and a half million people on our Web site or forty-four million?"
The first time you ran into this nest of nettles, you hopped over to the white board and cheerfully explained all about
cookie deletion
cookie blocking
multiple machine browsing
multiple browser browsing
multiple people on the same cookie
non-human traffic
dynamic IP addressing
page caching
javascript loading
called pixel placement
You didn't even get to the good stuff about comparing miles to gallons and how
different tools using
different date cut-off routines and
different methods to capture
different types of data to store in
different kinds of databases with a
different method of data cleansing and
different slicing and dicing segmentation to produce
different kinds of reports that ended up in
different feed for integration into
different datawarehouses
...before you were thanked for your help and shown the door -- permanently.
You don't fall for it this time.
This time you explain that the world of online marketing has been suffering from an delusion of precision and an expectation of exactitude.
You tell them that we live in a world of statistics and probabilities. We can't count all the stars in the sky, so we don't try. We don't try to get an actual count of
television watchers
radio listeners
magazine readers
billboard readers
bus poster readers
floor sticker readers
airline ticket jacket readers
sandwich board readers
Instead, we count some and estimate the rest.
You share the good news that we can do this better than any of the above -- and we've got some astonishing tools and techniques for dynamically targeting the audience and optimizing each one's experience.
You say, "We get 36.3 million people coming to our Web site."
The CMO lowers her half-glasses and gives you the look you last saw when caught using the office copy machine for party invitations. So you add, "With a 4% margin of error and it's a benchmark we can compare month over month from now on."
"So somewhere between 34 and a half and 38 million," she says.
"Pretty much right between them, in fact."
Disparagingly, she asks, "You really can't give me a more accurate number of how many people saw this digital marketing masterpiece that costs me tens of millions a year?"
"I can tell you whether our digital visitors are more engaged with our brand, come back more often, buy from us and discuss our products with their friends. How many people buy our products who saw our ads on CNN and 'Oprah' that cost you hundreds of millions a year?"
The VP of Advertising makes himself visibly smaller.
"I came here to show you a way that could save four million dollars of search marketing while boosting online sales by 6 to 8%," you say.
The scowl leaves the CMO's face. The odor of dubious data dissipates. Her eyes narrow as she leans forward and says, "Show me."
The numbers don't have to be precise -- just compelling.
Although I am not on the client side, I LOVED this article! It gave me some insight on the client side pain points, but also helped me think more strategically when positioning my website. Thank you.
Hear, hear!
Brilliant! I have been a marketing exec in the web space for years and a long-time subscriber to MediaPost newsletters. This is a brilliant post and probably one of the best articles on this subject I have ever read. Nice job, Jim!
Thank you (!!!!!).
I love this. Wonderful insight and comparisons.
great post! thank you!
"we live in a world of statistics and probabilities. We can't count all the stars in the sky, so we don't try"
Awesome!
Great post! I hate the dubious numbers game that Marketers play. But in reality it is impossible to get exact measurements, which sadly lead to much inflating of numbers by biased interests. I leave a few websites up in my browser tabs all day long, but yet spend maybe 20 minutes on each a day...does that Website claim 8 hours or 20 minutes? But when you deal with the CFO all they care about is did they get sales! That is the goal anyway.
Sometimes the best ideas/lessons/thought-provoking points to be made, come from stories & analogies. Great, great story here!
Diagramming Cookie Deletion and Dyn IP Addressing for the CMO??? Too funny...!
The CMO and the VP of Advertising return to their offices and immediating begin to dissemble. The nearest competitor hires away the R&D team, and a new technology is introduced that cuts the cost of doing what your client does by 20%.
You return to yours and try to figure out precisely 'how' you are going to deliver 6 - 8% when the GDP plunges 10%, and choose between Oprah and CNN when she is boycotted by whackos and CNN discovers that it's become an entertainment/talk channel instead of news.
As the old showbiz saying goes, "Nobody knows nothin'." Probablistic stats are better than nothing but not worth much without good analytics and informatics that can tell you "which" of your decisions are working.
Wonderful column - it sums up nicely the double-edged sword online created for itself years ago that still plagues us. Often when clients get stuck in the metric details we have to make them step back and say "Tell me, how are you measuring your other media?"
Then we get into the discussion of determining how we helped them sell product and get butts in seats - because really, that is what matters.
Thank you!
What a great article. I agree with Holly - it's not everybody who can deliver a barrel-full of laughs AND profound wisdom in under 1000 words. Nice job!
Stands and applauds. Bravo Jim!
That is simply the best piece I have read on online metrics in ages, if not ever (apologies to Josh Chasin who also "gets it" and writes brilliantly as well).
May I have permission to use:
* "the world of online marketing has been suffering from an delusion of precision and an expectation of exactitude"; and
* "we can't count all the stars in the sky, so we don't try"
down here in Australia when called on to explain the self-same situation.
great post.
i think the giant problem with reconciling today's metrics, whether they come from the web or advertising or pr or other functional discipline, is that we still (apart from direct-response print or tv that makes no use of any other medium) don't know how each of these tactics -- individually or collectively with others -- affects a customer's perceptions or purchase behavior.
i may see a print or tv ad, follow it up by going to a website, and not do anything about it for weeks or months until i overhear a conversation between two people on a plane who're talking about the product or the brand .... and i may STILL not do anything until much, much later, when i actually DO buy whatever it is, but have completely forgotten about how i came to know about the thing. so even my response on a survey asking me "how did you hear about us" is suspect, because i will probably lie to avoid looking like an idiot.
this is why i personally favor net promoter score as the key metric that should be tracked above all others ... we do know, statistically, that people are more often likely to trust the recommendations of family/friends/colleagues, and we can go about getting NPS data in a fairly straightforward way. not as fun as explaining cookie deletion and dyn ip to the cmo ... but probably much more worthwhile, in the long run.
Jim..
Wow was this ever good. It's just so refreshing to see this kind of thinking. Thank for a great post. I've been making this point to clients for so many years - and it's just great to have the story so well told.
Yes! Now read the White Paper on C3 Metrics which solves this. http://c3Metrics.com
So typical of a CMO, wanting to track and measure every possible metric across a variety of tools. It should be established in the beginning that one source will be used as the benchmark.
Great post Jim!
Eric
http://onlinemediaanalyst.com
I am absolutely delighted and humbled by all of your comments. Thank you so much. I'm speechless... until the next <a href="http://www.emetrics.org">eMetrics Marketing Optimization Summit</a> ;-)
Great stuff, Jim. The problem is that the Ad and PR Agency reps are selling the CMO on "eyeballs" and the web metrics guys are delivering revenue. Which is the CEO going to buy?
Jim, Great post which has universal approval from all those that are actually working at the coal face of on-line marketing. Don't remain speechless for too long.
Brewster