Newsflash: Biz Media Again Fooled By Bogus Research

by , May 11, 2010, 2:15 PM
  • Comment (8)
  • Recommend (49)
Subscribe to Metrics Insider

I continue to see a number of highly intelligent business editors covering the marketing industry get fooled into running stories based on PR polls disguised as research. And when that happens, everyone in the marketing community gets hurt.

New research is exciting and relevant to marketers. It leads to new thinking and ideas. And it makes good copy.  As an example, one major trade publication recently ran a story about a survey of "Chief Marketing Officers" and marketing measurement with the headline "Survey Finds Marketing Contributes to the Bottom Line."  This undoubtedly made it into countless PowerPoint presentations in budget meetings.

 But scratch beneath the surface and this "survey" was comprised of 423 "business executives and marketing professionals" who may have come from any industry, or be in companies ranging from billion-dollar marketing budgets to $10k budgets, or may be CEOs or entry-level managers. In other words, unlike what its title indicates, the sample is reflective of no particular group, aside from those willing to be surveyed.

 Even more dangerously, the story went on to state that 39% of the respondents agreed the marketing is doing a good job of contributing to the financial condition of the business, up from 19% last year. Presumably this was referring to better use of marketing metrics to better measure marketing. But were these the same people surveyed last year? Were they selected to match the profile of people surveyed last year, so their responses would be scientifically valid?  No. They were just this year's batch of willing respondents, bearing little resemblance to last year's group, thereby making any sort of trend analysis invalid. 

Unfortunately, this is neither uncommon nor harmless.  Marketing struggles every day to earn trust and credibility with finance, operations, sales, and other functions that have a more skeptical and discerning eye when it comes to research. But if the marketing media suggests it's OK to accept PR polls as research, it indirectly encourages marketers to include some of these "findings" in rationalizing their recommendations to others.

 In fairness to my hard-working friends in the media, most editors and staff writers (let alone marketers themselves) have not had the benefit of training in how to tell a bogus survey from a truly reliable one. They're very busy trying to produce more content to feed both online and offline vehicles with smaller payroll and more pressure to get readers. So perhaps I can offer a few simple tips to separate the fluff from the real stuff:

 1.              Before you even read a survey's findings, ask to see a copy of the survey questionnaire and check out the profile of the respondent group so you know how to interpret what you're being told. Get a clear sense of "who" is supposedly doing/thinking "what," and ask how the respondents were incentivized. Then ask yourself if a reasonable person would really put the effort into answering completely and honestly.

 2.              Check the similarity or differences among survey respondents. If the vast majority of them share similar traits (e.g. company size, industry group, annual budget), then it's fair to extrapolate the findings to the larger group they represent. But if no single characteristic (other than being in "marketing") ties them together, they represent no one, regardless of the number of respondents. You'll need to separate the responses by sub-group like larger marketers versus smaller ones, or B2B vs. B2C. In general, you'll need at least 100+ respondents in any sub-segment to make it a valid result. 

3.              Check to see if the sample has been consistent year-over-year. If it has, you can safely say that something has or hasn't changed from one year to the next. But if the sample profile is substantially different year-to-year, comparisons aren't valid due to differences in the perspectives or expertise of those responding. 

4.              Ask about the margin of error. Just because 56% of some group say they feel/believe/do something, doesn't mean they actually do. EVERY survey comes with a margin of error. Most of the PR-driven polls in the marketing community use inexpensive data collection techniques that offer no ability to validate what people say and no mechanisms to keep respondents honest. Consequently, 56% may actually mean somewhere between 45% and 65%  -- which may change the interpretation of the findings.  Ask the survey sponsor about margin of error. If they aren't sure, don't publish the numbers.

 And if all that is too difficult, call me and I'll give you an unbiased assessment for free.

 It's difficult to produce a survey that provides real insight and meaningful information. That's why real research costs real money. And while there's nothing wrong with polls being used to gather perspective from broadly defined populations, or PR folks using these for PR purposes, confusing PR with real research slowly poisons the well for all of us in the marketing community.

0 comments on "Newsflash: Biz Media Again Fooled By Bogus Research".

  1. Kevin Horne from Lairig Marketing
    commented on: May 11, 2010 at 2:27 p.m.

    Pat:

    I like that you put "Chief Marketing Officers" in quotes. The CMO Council itself won't even admit how few of its survey respondents are actual CMOs.

    And when you think of it, if real CMOs had so much free time on their hands to be taking part in dopey surveys every week, then no wonder so many companies' marketing programs are such a mess...

  2. M Cohen from marshall cohen associates
    commented on: May 11, 2010 at 2:32 p.m.

    You are right......most editors and staff writers will publish anything and everything that comes across their desks.....alas, I fear, you are trying to teach a pig to sing. It's hard to do and it annoys the pig.

  3. Paula Lynn from Who Else Unlimited
    commented on: May 11, 2010 at 2:47 p.m.

    Waste, waste, waste.

  4. Carolyn Hansen from Hacker Group
    commented on: May 11, 2010 at 5:25 p.m.

    This is one of my biggest pet peeves. No one thinks critically about these things. Thanks for pointing it out.

    Mark Twain or Disraeli or somebody really clever once said, "There are three kinds of lies: Lies, damned lies and statistics."

  5. John Grono from GAP Research
    commented on: May 11, 2010 at 6:04 p.m.

    Hear, hear Pat.

    In this digital age of open access people confuse a tool with a skill. Down here in Australia it is mightily unpopular to put one's head out of the bunker and criticise this disturbingly more frequent practice, so I applaud your courage.

  6. Jan Rabinowitz from January Consulting
    commented on: May 11, 2010 at 7:05 p.m.

    Pat,
    While you do make a good point, as a former researcher for a major trade magazine publisher, I must point out that some of the large publishers do employ professional researchers to conduct their surveys. I know that my former employer, Penton Media (the former Prism Business Media/Primedia portion), made it a point to use valid statistical samples. We adhered to the ethics statement put out by ABM regarding research. Our reports/questionnaires, etc. were always available to anyone who requested them and we encouraged the editors to include methodology information in the articles.

  7. Laura Patterson from VisionEdge Marketing
    commented on: May 11, 2010 at 9:54 p.m.

    Pat this study followed well practiced research methodologies - the publications had the complete report and they chose what they wanted to write in their stories, whether that is what we would have wanted published is not in our control, that's the prerogative of editors. So while you may have some good points regarding studies you picked a study that didn't represent the points you made.

  8. Scott Turner from GfK MRI
    commented on: May 12, 2010 at 10:40 a.m.

    Lots of Data being generated every minute, but Data does not equal Good Research

Leave a Comment

Sign in to leave a comment. Don't have an account? Join Now

Recent Metrics Insider Articles

» Metrics Insider Archives