• by John Bremer , November 19, 2010

Among the wide-ranging -- and occasionally unusual -- job titles I've had in my career, almost all have featured a single word: "representativeness." Confusing to some and misunderstood by others, "representativeness" appears in these job titles because of its fundamental ability to help the research industry answer our client's most pressing business questions. 

Despite this essential role, not everyone in the research industry can reliably define representativeness. Most researchers worth their salt, along with the more experienced senior marketing officers, can speak generally to the concept and maybe connect it to broader sampling methodology. For the most part, however, the term remains remarkably elusive for most business people, even those whose day-to-day responsibilities are directly impacted by research results.

So I pose the question: What is representativeness, and how do we know it when we see it? As someone with a long job history of positions with "representativeness" in the title, I hope I can bring some perspective to this question.

In essence a general understanding of a representative sample is that it "looks" like the population of interest, and the results can be used to "accurately" project the finding from the sample to the population. Even the most experienced researchers in the field, however, might take issue over whether a sample is truly representative. 

As one example, I spoke at a conference on sampling theory attended by leading experts in the field, after which I participated in a working session focused on improving some well known government-sponsored surveys. One particular survey illustrates the difficulty of identifying and defining representativeness.

The survey in question had been undertaken primarily to understand a single variable, and this variable was showing some unexplainable changes from the previous year. The sample had been correctly drawn, the survey carried out with precision and the end data weighted to ensure that everyone had an equal probability of selection. Theoretically, the result should be "representative" -- but in reality this was not the case at all. Indeed, a quick view of the demographic and behavioral data showed some differences in the characteristics of the sample as compared to the population. After correcting these differences, the trended results for the variable of interest matched what was known to be true of the population. We had secured representativeness.

In this case, we were able to pinpoint the underlying problem due largely to the experience of several participating non-academics at the conference - myself included - whose day-to-day roles rely on empirical evidence in addition to theory, and thus had an advantage in diagnosing the disconnect. Nonetheless the exercise made me think of the elusive, and often widely misunderstood, definition of representativeness. If experts like those assembled had problems understanding how to diagnose whether results were representative, I thought, how can those with less experience do so?  The following simple rules should help every time.

·    Have a very precise statement regarding your population of interest.  Often, when I ask what the population of interest is for a study, I hear the words "general population"  -- only to discover that there are many more qualifying criteria. Even if you think your population of interest is general population, be more specific (e.g. "U.S. adults, aged 18 or older").

·    Know what your specific population of interestlooks like with regard to its demographic characteristics and, if possible, know some non-demographic characteristics of the population - even if you have to gather that information from other surveys of the same population for comparison purposes.

·    Understand where your sample is coming from.  If it is in some way targeted, or gathered in a way that excludes parts of your population of interest, understand how the vendor intends to make the sample representative.

·    Understand how the questions of interest are being asked.  If the concept you are trying to measure is not being investigated correctly, the result will not be representative - even though the sampling methodology may be.

·    Understand how the result is being weighted or normalized.  If too much weighting is done, your measures may not be stable and "weird" results may occur.

·    Finally, if everything about the process seems like it works, look at the data empirically.  Do the demographics of your sample line up with the population?  Even more importantly, for diagnostic purposes, do non-demographic measures of your sample compare to what you know about the population?  Make sure your sample design gives you metrics for determining if the result is representative.

These tips hold whether we are talking behavioral clickstream, survey data or even large scale qualitative. Keep in mind that theory is important in creating a representative sample, but empirical evidence is the final confirmation. From someone whose job title seems to include "representativeness" at every stage of my career, I can say these steps will help ensure a truly representative sample -- and therefore better results for the businesses that rely on them.