Sampling Error: The Distorted Reality Of Research

When we look in the mirror, it’s easy to see not only the physical attributes that define our unique features, but also the layers beneath that makeup who we are — those tired eyes given as a gift of raising your children, the cultural mark that ties you to your ancestors, or that scar of survival you proudly wear as a badge. 

Now imagine looking into the mirror and seeing something stare back at you that is a superficial version of yourself. You only see bits and pieces of the person you know yourself to be, but never your entire reflection. This is the “sample crisis.” 

How do we add pigment to traditional research surveys' shades of gray in today’s diverse world? Our culture demands it, and our antiquated research questions are suggesting that it’s time to change the way things have always been done. This year’s SampleCon played host to many concerns in that area, validating the unease amongst the global research professional community. 



Beyond the age-old challenges in market research (retaining quality, shortening surveys, keeping panelists engaged) is a more pressing issue of capturing diverse voices. Perhaps the root of keeping panelists engaged is deeper than the look and feel of the survey platform. Perhaps we are not connecting to people how it matters most. 

To better understand where we are missing the mark, we need to look to the next generation of respondents. We know that, more than any previous generation, Gen Z is deeply rooted in their identity. 

While most agree that the way we ask demographic questions isn’t ideal, there is still a lot of debate about changing it. There are many considerations researchers have to make when changing such fundamental things, like how we ask gender, age, race, and ethnicity questions. 

Do we need to make historical comparisons? Do we need to have a comparable demographic makeup to other existing client research? How do we ensure we’re inclusive and still able to accurately analyze the data? 

We know that humans are complex. Very few of us identify as simply as the standard set of demographic descriptors often included in surveys. However, the considerations above can be paralyzing when we’re all seeking THE right answer. 

Recently the following question was asked to me: “How do you define the panelist of the future?” 

My response: “Fluid and evolving.” 

That’s to say, there is no perfect answer right now. But we can’t remain stagnant while we wait for the perfect answer to come along. Instead, we must focus on “better” versus “perfect.” 

It’s vital to take action by auditing your own practices. Evaluating what we can change today. As a first step, define standard screeners and demographic questions through a lens of inclusivity. Reimagine and broaden how we ask questions like gender identity in a nonbinary way and age with a list that doesn’t stop at 65-plus. 

Shift from the traditional gender question with two options (male and female) to an open-ended question that asks a person’s gender identity and provides a more inclusive set of options, including Man, Woman, Transgender, and non-binary. To avoid placing people into an “Other” box, we change the language to allow a respondent to select “Prefer to self-describe.” 

It’s a small example of the seismic shift needed. It’s not perfect, but if we’re able to bring a bit more humanity into the here and now, it’s a start.



4 comments about "Sampling Error: The Distorted Reality Of Research".
Check to receive email when comments are posted.
  1. Ed Papazian from Media Dynamics Inc, July 26, 2022 at 1:12 p.m.

    Very interesting and many good points, Kelcey.

    What I see in so many of the studies being conducted online these days is far too little consideration for the need that respondents have to really understand the questions and think out their replies. Instead, the designers of too many of these studies try to cram as much as they can into a design which accepts that the respondent will not spend too much time in providing answers---due to the lesser attention spans inherent in  online research. Then, they trot out that old canard---'sampling error"--- to create the false impression that the findings are to be accepted as if one can calculate the accuracy of any survey merely by knowing its sample size. Sorry---you can't.

  2. John Grono from GAP Research, July 26, 2022 at 10:22 p.m.

    Very interesting approach Kelcey.

    The classic definition of research is that the client wants it yesterday, cheap and accurate.   The classic response is that research is like a three-legged stool - you need all three and if you have just two it all falls over.

    Of the three parameters, 'accurate' appears to be the casualty.   Or as Meat Loaf put it ... two out of three ain't bad ... seems good enough these days.

    But as time passes, things like representativeness of online surveys shouldn't be much of a problem these days, as virtually all person-based surveys suffer from the same issues.   However, IMHO the bombardment of online surveys is exacerbating the problem as more-and-more people just bail out.   Online surveys are generally based on respondent quotas.   There are some upfront questions to try to gauge representativeness but they basically end up as indicators of respondent skews which are 'magically' solved by re-weighting (which more times than not can exacerbate the skew).

    Second, the questionnaires are often poor or can't be accurately responded to because the 'bail-out' option of "Don't Know" is too often omitted.   My favourite is the 'follow-up' questions of "And why did you give that response?", as they try to dig out some possible research gems that they hadn't considered (cough,cough).   [My favourite response is "I gave you that response because you asked so nicely".]

    As time frames and budgets shrink we're in a situation where 'more meansless'.

  3. Kelcey Curtis from VMLY&R replied, August 9, 2022 at 6 p.m.

    Thanks, John. It's so critical that we're putting the people taking the time to participate in our research first. Hopefully we're exploring more ways to ensure our samples are diverse and inclusive so we can rely less on statistical tactics to create representativeness. And I think we can all agree that when accurate, fast, and inexpensive is our remit, it can certainly be tough to navigate. 

  4. Kelcey Curtis from VMLY&R replied, August 9, 2022 at 6:04 p.m.

    Thanks for sharing, Ed. It's so important that we remember that real people are on the other side of our research. Surveys are our way to have conversations with real human beings. While we'd ideally be able to tackle every research question in one study, your point about fatigue is an important one to keep in mind. 

Next story loading loading..