Commentary
Sampling Error: The Distorted Reality Of Research
- by Kelcey Curtis , Op-Ed Contributor, July 25, 2022
When we look in the mirror, it’s easy to see not only the physical attributes that define our unique features, but also the layers beneath that makeup who we are — those tired eyes given as a gift of raising your children, the cultural mark that ties you to your …
4 comments about "Sampling Error: The Distorted Reality Of Research".
Read more comments >
Next story loading 
Very interesting and many good points, Kelcey.
What I see in so many of the studies being conducted online these days is far too little consideration for the need that respondents have to really understand the questions and think out their replies. Instead, the designers of too many of these studies try to cram as much as they can into a design which accepts that the respondent will not spend too much time in providing answers---due to the lesser attention spans inherent in online research. Then, they trot out that old canard---'sampling error"--- to create the false impression that the findings are to be accepted as if one can calculate the accuracy of any survey merely by knowing its sample size. Sorry---you can't.
Very interesting approach Kelcey.
The classic definition of research is that the client wants it yesterday, cheap and accurate. The classic response is that research is like a three-legged stool - you need all three and if you have just two it all falls over.
Of the three parameters, 'accurate' appears to be the casualty. Or as Meat Loaf put it ... two out of three ain't bad ... seems good enough these days.
But as time passes, things like representativeness of online surveys shouldn't be much of a problem these days, as virtually all person-based surveys suffer from the same issues. However, IMHO the bombardment of online surveys is exacerbating the problem as more-and-more people just bail out. Online surveys are generally based on respondent quotas. There are some upfront questions to try to gauge representativeness but they basically end up as indicators of respondent skews which are 'magically' solved by re-weighting (which more times than not can exacerbate the skew).
Second, the questionnaires are often poor or can't be accurately responded to because the 'bail-out' option of "Don't Know" is too often omitted. My favourite is the 'follow-up' questions of "And why did you give that response?", as they try to dig out some possible research gems that they hadn't considered (cough,cough). [My favourite response is "I gave you that response because you asked so nicely".]
As time frames and budgets shrink we're in a situation where 'more meansless'.
Thanks, John. It's so critical that we're putting the people taking the time to participate in our research first. Hopefully we're exploring more ways to ensure our samples are diverse and inclusive so we can rely less on statistical tactics to create representativeness. And I think we can all agree that when accurate, fast, and inexpensive is our remit, it can certainly be tough to navigate.
Thanks for sharing, Ed. It's so important that we remember that real people are on the other side of our research. Surveys are our way to have conversations with real human beings. While we'd ideally be able to tackle every research question in one study, your point about fatigue is an important one to keep in mind.