| ||||||||||||
PRINT
COMMENT
SUBSCRIBE
RSS
In total, the survey panel reported a mean response of 27% when asked, "on a scale of 0% to 100%, how many marketers suck?"
This has been a test of the emergency BS system. Had this been a real, scientifically based survey, you would have been instructed where to find the nearest bridge to jump off.
Actually, it was a "real" "survey." I found five teenagers in a local shopping mall loitering around the local casual restaurant chain and asked them a few questions. Seem valid?
Of course not. But this one was OBVIOUS. Every day we marketers are bamboozled by far more subtle "surveys" and "research projects" which purport to uncover significant insights into what CEOs, CFOs, CMOs, and consumers think, believe, and do. Their headlines are written to grab attention:
-- 34% of marketers see budgets cut.
--71% of consumers prefer leading brands when shopping for .
And my personal favorite:
-- 38% of marketers report significant progress in measuring marketing ROI, up 4% from last year.
Who are these "marketers"? Are they representative of any specific group? Do they have anything in common except the word "marketing" on their business cards?
Inevitably such surveys blend convenience samples (e.g. those willing to respond) of people from the very biggest, billion-dollar-plus marketers to the smallest $100k annual budgeteers. They mix those with advanced degrees and 20 years of experience in with those who were transferred into a field marketing job last week because they weren't cutting it in sales. They commingle packaged goods marketers with those selling industrial coatings and others providing mobile dog grooming.
If you look closely, the questions are often constructed in somewhat leading ways, and the inferences drawn from the results conveniently ignore the statistical error factors that frequently wash away any actual findings whatsoever. There is also a strong tendency to draw conclusions year-over-year when the only thing in common from one year to the next was the survey sponsor.
As marketers, we do ourselves a great disservice whenever we grab one of these survey nuggets and embed it into a PowerPoint presentation to "prove" something to management. If we're not trustworthy when it comes to vetting the quality of research we cite, how can we reasonably expect others to accept our judgment on subjective matters?
So the next time you're tempted to grab some headlines from a "survey" -- even one done by a reputable organization -- stop for a minute and read the fine print. Check to see if the conclusions being drawn are reasonable given the sample, the questions, and the margins of error. When in doubt, throw it out.
If we want marketing to be taken seriously as a discipline within the company, we can't afford to let the "marketers" play on our need for convenience and simplicity when reporting "research" findings. Our credibility is at stake.
And by the way, please feel free to comment and post your examples of recent "research" you've found curious or questionable.
14 people recommend this article.
RSS FEEDS
And it's not just marketing issues--political pollsters are notorious at this sort of thing.
As the saying goes, "If it's worth doing, it's worth doing well." To which the corollary is "If it's not woth doing, it's not worth doing well."
Reality, anyone?
http://www.marketingprofs.com/8/online-research-traps-derail-marketing-strategy-horne.asp
Unfortunately, the ease of access and use of the internet for 'research purposes' has seen a torrent of such abuses of the word "research" here down under.
Paul - I was going to cite the "not everything that can be measured should be", but your beat me to it. David - thank you for alerting me to the Wall Of Shame website, it will get great usage here.
My personal favourites are:
(i) a VERY real example from around 5 years ago when online research was conducted to measure the advertising effectiveness of a campaign for an existing bread brand that was being relaunched. The research company used online for cost reasons, but failed to realise that the bread brand was only in two of Australia's six states - no wonder the advertising and brand awareness plummeted!
(ii) a 'research project' we had to conduct when I was at university. The objective was to prove that all men beat their wife. Solution: "When was the last time you beat your wife?" along with a series of time lines that ended with "more than 10 years ago" and removing the "no answers". Moral of the story - give me the answer and I can give you the question (but I would never take it to field!).
(iii) back when we had two daily newspapers in our city an advertising brawl was on as to which classifieds dominated the car market. The headline read "We sell 73% more cars.*" In the small print the asterisk went on to say "... than commercial vehicles".
Remember these few sage-like adages:
1. Not everything that can be measured, should be.
2. FALSE assumption #324: almost all people almost all the time make choices that are in their best interests or at least better than the choices made by third parties.
3. Efficiency is relevant, but it's not the only goal.
4. All human stories are subject tto errors of omission, fact or interpretation, regardless of intent.
Bravo, ole', Pat. More. More.
If you want to see more examples of bad research check out the Research Wall of Shame here . http://www.researchwallofshame.com/
David Slatter
Average incomes are ludicrous. Some simple math shows that all it takes is one billionaire to seriously throw the curve. That's why in any serious looks at income, you look at MEDIAN income, not average income.