• by February 29, 2008

As a self-confessed, card-carrying member of the revolutionary "Free the Data" militia for the last nine years, I read  Mike Bloxam's latest TV Board column with a smile. I'll be the first to confess that set-top box data is not perfect. Not every television in every cable/satellite/telco household has a set-top box. In a privacy-compliant data set, no demographic data is attached to the tuning information other than the zip code associated with the account. The set-top box is blind with respect to which person(s) may or may not be watching. While these and other issues are unquestionably important, I believe they are all resolvable with high quality, granular tuning data and sound mathematics. We may disagree, but we should be able to debate the issue. I'll take the set-top box data, you take your silly little panels and we'll meet on the playground after school and duke it out. Ah, if it were only that easy. The problem is not the set-top box data, Mr. Bloxam. You are grossly mistaken.

As Ms. Scheppach is now acutely aware, set-top box data and its proponents are routinely vilified. Research based on small, opt-in panels is held up as the gold standard, but the approach has severe limitations which are often ignored. For example:

(1) We currently recruit and routinely pay households to participate in panel research. The recruiting process introduces a confounding bias. Are people who agree to have all of their television viewing habits monitored typical of the population as a whole? Does payment to participate in the study affect viewing behavior? Does the fact that panelists know they are being monitored affect viewing behavior? To put it more bluntly, does wiring a panelist's home in "Truman Show" fashion truly capture typical media consumption? I think not.

(2) I am ashamed to admit that as an industry we often feel fortunate to get one in four recruited households into the panel -- the three that refuse contribute to another bias. If those who are not successfully converted share certain characteristics and those characteristics affect television viewing behavior, those effects are completely lost on the panel.

(3) Once we manage to get a household in a panel, we ask the members (and guests) to alter their viewing behavior and consciously punch buttons on a people meter. This required ancillary behavior introduces more bias and error.

(4) When we don't get the demographics right in the panel, we apply weighting to the data which further degrades the research.

As a cursory look reveals, recruited panels are far from ideal and there is nothing inherently wrong with that. But the research based on such panels is often presented as perfect. Case in point: the 32 million people who watched the Oscars on Monday, the smallest audience in years. It must be true; I read it in The New York Times.

And that, ladies and gentlemen, brings me back to the issue at hand. Arguing about which data should or should not be used is missing the point. The research is what matters. How many people watched and who was glued to the tube? That's what everyone wants to know. The problem is not in the data -- the problem is in the reporting. And while the reporting has many flaws, let's talk about the most egregious. At what point did it become acceptable for a research company to produce audience estimates that have no error margins or confidence intervals? Is research relevant without context?

Unfortunately, I don't think anyone anticipates the MRC, ANA or the AAAAs forcing error margins and confidence intervals onto television audience ratings any time soon. So until then, Mr. Bloxam, the only way the industry can validate if 32 million people really did tune in to the Oscars on Monday is to look at the set-top box data. Media buyers like Ms. Scheppach and their clients fund the ad-supported media; forgive them if they don't take you and your small sample's word for how much they should be paying. Besides, everyone deserves to be counted, not just the hand-selected, weirdo, voyeur-loving few who think $30 a month is fair compensation for their own little "Truman Show" fantasy.