• by Ted McConnell , Featured Contributor, January 7, 2021

The recent election and pandemic have prompted the usual commentary from marketers pronouncing that surveys don’t work. But they do work. Surveys are tools, and like any tool they have to be used for the right job, by people who know how to use them.

It’s worth talking about, because gaslighting the world’s most tried and true research methodology is damaging. Truth is fragile, and confidence in information is essential to making great decisions. 

Survey methodology provides unique insights. Surveys are the only reliable tool we have as a society to tell us what people are thinking. It’s safe to say that every existing major brand became a major brand through consumer understanding gained, at least in part, via survey research.

Honestly conceived and professionally executed research is vastly more right than wrong, but nothing is perfect.

As it is with all tools, there’s a right one for the job.  Neuroscience, for example, detects emotional reactions reliably, but the person must be observed.  “Big Data” is great, but data is generally a record (an observation) of what people do (not what they think), and is progressively harder to get due to privacy regulation.  Surveys can tell us what people think, how they feel, and how they act.

Looking Under The Covers

Survey quality (or the perception of quality) can suffer for any number of reasons:

1)    Research providers are constantly cutting costs as a way to respond to competitive RFPs from corporate buyers. Relentless cost-cutting always impacts quality.  Companies with strong internal research organizations prosper long-term. Companies who treat research as a commodity don’t.  

2)    Often, marketers are not really looking for the truth. There are no methodological trick to overcome this issue. When research sets out to validate rather than discover, the results are broken on the drawing board. “Research proves my product is better” has reached the point of being a joke. In that game, everybody loses long-term: consumers, brands, and researchers.

3)    Survey data can be good while projection is flaky. Projecting the opinions of a few people onto entire populations gets tricky. Research people spend a huge amount of time trying to make sure the right subset of a population has been sampled. There will always be error, but it is well quantified, so can easily be incorporated into a good decision. Projectability is at the core of the science behind good surveys.

4)    The most common target of critics is the fact that many surveys are taken by “survey takers” -- people who take surveys for pocket money. This is a valid criticism, but to give the short answer, there are millions of these survey takers. There is sufficient oversight, and diversity in people who are willing to answer surveys, that the right people can be reliably selected for most surveys. Someone who gives their opinion for money has a very good reason not to lie. If they get caught, they lose money.

5)    Surveys methods are too accessible.  Any fool can ask four friends a question and claim they administered a survey. The public perception is that it’s simple, so it should be cheap. Casual surveys are fine for casual questions, but having to make high-stakes decisions means the survey should be well-designed in every way.

6)    There is a litany of issues endemic to asking people questions. People lie, for example. But professionally done surveys catch liars and throw out their results. Sometimes people don’t read the question, but it’s easy to catch them. Survey programmers can write poorly constructed questions, but oversight can easily fix this.

It’s straightforward to get the truth if you want the truth, but it’s not easy to undo the costs of a bad decision.

Lack of trust in research causes results to be ignored by decision-makers,  in turn causing big damage like failed initiatives, bad policy, and damaging public perceptions (like “Masks don’t work”). The absence of trusted information throws the advantage to autocrats in the C-suite and in the White House. People who make their own truth surely don’t want research to work.

What can you do? Before you pay for research, make sure you are using the right tools, take the time to understand the limits of the tools, and frame your questions in a neutral way so you don’t bias the outcome.

Given its frailties, it’s easy to see why survey research is maligned. However, most of the issues are not issues with survey methodology. The issues are with people: their expectations, their desire to be right, their need to save money, and their ambition.