Commentary
Survey: 27% Of Marketers Suck
- by Pat LaPointe , Op-Ed Contributor, May 5, 2009
In total, the survey panel reported a mean response of 27% when asked, "on a scale of 0% to 100%, how many marketers suck?"
This has been a test of the emergency BS system. Had this been a real, scientifically based survey, you would have been instructed where to find the nearest bridge to jump off.
Actually, it was a "real" "survey." I found five teenagers in a local shopping mall loitering around the local casual restaurant chain and asked them a few questions. Seem valid?
Of course not. But this one was OBVIOUS. Every day we marketers are bamboozled by far more subtle "surveys" and "research projects" which purport to uncover significant insights into what CEOs, CFOs, CMOs, and consumers think, believe, and do. Their headlines are written to grab attention:
-- 34% of marketers see budgets cut.
--71% of consumers prefer leading
brands when shopping for .
And my personal favorite:
-- 38% of marketers report significant progress in measuring marketing ROI, up 4% from last year.
Who are these "marketers"? Are they representative of any specific group? Do they have anything in common except the word "marketing" on their business cards?
Inevitably such surveys blend convenience samples (e.g. those willing to respond) of people from the very biggest, billion-dollar-plus marketers to the smallest $100k annual budgeteers. They mix those with advanced degrees and 20 years of experience in with those who were transferred into a field marketing job last week because they weren't cutting it in sales. They commingle packaged goods marketers with those selling industrial coatings and others providing mobile dog grooming.
If you look closely, the questions are often constructed in somewhat leading ways, and the inferences drawn from the results conveniently ignore the statistical error factors that frequently wash away any actual findings whatsoever. There is also a strong tendency to draw conclusions year-over-year when the only thing in common from one year to the next was the survey sponsor.
As marketers, we do ourselves a great disservice whenever we grab one of these survey nuggets and embed it into a PowerPoint presentation to "prove" something to management. If we're not trustworthy when it comes to vetting the quality of research we cite, how can we reasonably expect others to accept our judgment on subjective matters?
So the next time you're tempted to grab some headlines from a "survey" -- even one done by a reputable organization -- stop for a minute and read the fine print. Check to see if the conclusions being drawn are reasonable given the sample, the questions, and the margins of error. When in doubt, throw it out.
If we want marketing to be taken seriously as a discipline within the company, we can't afford to let the "marketers" play on our need for convenience and simplicity when reporting "research" findings. Our credibility is at stake.
And by the way, please feel free to comment and post your examples of recent "research" you've found curious or questionable.
Excellent points. And not limited to marketers; look at some of the "surveys" eagerly adopted, practically word for word, and reprinted by lazy journalists with neither the time nor inclination to check the stories/news releases further. A little healthy skepticism would go a long way towards bolstering credibility.
Poorly designed and reported surveys is one of my major pet peeves -- and a regular topic I post about. Most recently I ranted about a Brandweek article that purported to tell me about the increasing importance of event marketing . . . using a survey sponsored by the Event Marketing Institute. Sheesh. You don't need much of a BS detector to figure how that's going to come out.
I really enjoyed reading this and getting a good laugh this morning. You're absolutely right. I've seen people spend thousands of dollars on research and they still don't have a clue about what's happening with their target markets. Numbers are being passed around but you can learn more from a small group talking to them one-on-one than you can from a bunch of inflated results where people are clicking through like test rats trying to get the pellet that comes out of the shoot.
Statistics minus context = tailored opinion
How many times have you witnessed marketers twisting research findings in a certain direction just to match their preconceived conclusions? Or worse yet, to match the creative direction they have already come up with because the creatives think "it would be so cool"? Drives me crazy. Solid marketing solutions come from real, unadulterated, unbiased research and information -- you can still be incredibly creative based on real facts.
Amen. Think tobacco industry.
Yesterday, I was surprised with the following headline in main newspapers and in prime time news: 7 of 10 argentinians fell they are successful
In a country with 30% of poors it was, at least, a little bit curious.
When I went into the article, I found the definition of being successful: "to have a good job and earn a lot of money"
My curiousity was increasing word by word but....
Later, I found the solution of the mistery.
For achieving such espectacular headline, they had interviewed 200 persons, in a range between 18 and 45 years old, in Buenos Aires city.
Argentina's population is around 40 millions and, although Buenos Aires is the capital city, there are more than 1000 cities among the country.
I think that it could be a good example.
This made me laugh and cry- I have seen my share of bad research over the years; often times done for completely the wrong reasons, and compounded by bad research methodology. It does not help that people use many of the free online survey tools with the mistaken belief that they can just do their own research.
If you want to see more examples of bad research check out the Research Wall of Shame here
. http://www.researchwallofshame.com/
David Slatter
Brilliant. Pat, I hope you popped something meanningful writing this. Love it -- and will forward to other so-called market-eers I know.
Remember these few sage-like adages:
1. Not everything that can be measured, should be.
2. FALSE assumption #324: almost all people almost all the time make choices that are in their best interests or at least better than the choices made by third parties.
3. Efficiency is relevant, but it's not the only goal.
4. All human stories are subject tto errors of omission, fact or interpretation, regardless of intent.
Bravo, ole', Pat. More. More.
Bogus research by the ton is what you get when you have opportunistic marketers and other communicators perfectly willing to take advantage of a gullible, unquestioning public who are no less impressionable than a bunch of 6-year-olds.
100% of industry insiders agree - "This post rocks!"
Pat I always enjoy your postings, but today you have written what is now my all-time favourite MediaPost posting.
Unfortunately, the ease of access and use of the internet for 'research purposes' has seen a torrent of such abuses of the word "research" here down under.
Paul - I was going to cite the "not everything that can be measured should be", but your beat me to it. David - thank you for alerting me to the Wall Of Shame website, it will get great usage here.
My personal favourites are:
(i) a VERY real example from around 5 years ago when online research was conducted to measure the advertising effectiveness of a campaign for an existing bread brand that was being relaunched. The research company used online for cost reasons, but failed to realise that the bread brand was only in two of Australia's six states - no wonder the advertising and brand awareness plummeted!
(ii) a 'research project' we had to conduct when I was at university. The objective was to prove that all men beat their wife. Solution: "When was the last time you beat your wife?" along with a series of time lines that ended with "more than 10 years ago" and removing the "no answers". Moral of the story - give me the answer and I can give you the question (but I would never take it to field!).
(iii) back when we had two daily newspapers in our city an advertising brawl was on as to which classifieds dominated the car market. The headline read "We sell 73% more cars.*" In the small print the asterisk went on to say "... than commercial vehicles".
This should be re-posted at least once a year. Maybe it will discourage the over-abundance of dodgy "research findings" based on dud methodologies and cherry-picked "analyses".
Congrats - The only survey you need to worry about is your own. Is your marketing working or not ??? That is the question which will start you on the correct path.
Cheers Kurt - Australia's Email Marketing Guru - http://www.kurtjohansen.com
Huzzah! This was the exact point of my marketing engineering class at the end of my MBA. I agree that you should repost this annually and send it to anyone who has "marketing" in his/her title, especially marketing researchers. It's shocking the lack of common sense employed when interpretting raw data!
my contribution to the topic, from an article i did for marketingprofs last July
http://www.marketingprofs.com/8/online-research-traps-derail-marketing-strategy-horne.asp
Sample representativeness counts!! How questions are asked is important!! Ethics are important!! All this and more we've known for ages--but need to point out over and over again. For which, we're branded purists!!
As the saying goes, "If it's worth doing, it's worth doing well." To which the corollary is "If it's not woth doing, it's not worth doing well."
Reality, anyone?
Thank you. Thank you. Thank you. I only wish that this were more broadly distributed. But I will do my best to spread it around. One of my favorite recent examples was a summary of 2 focus groups that used a bar chart to "quantify" the findings. And the client seemed to take it seriously!
And it's not just marketing issues--political pollsters are notorious at this sort of thing.
Love this. If your sample is only of people who responded from a Twitter post or from your client base then your results are bogus. Just saying.