• by Gary Holmes , December 6, 2011

For more than half a decade, many in the TV industry have looked forward to set-top-box measurement as an elixir that would deliver better TV ratings. The theory is that if Nielsen or another ratings company could extract viewing information from the millions of set-top boxes already installed in viewers’ homes, the larger sample size would deliver more accurate -- or at least more stable -- ratings.

To that end, in 2007 Nielsen created a separate business unit (then called Nielsen Digital Plus) to build a set-top-box business.  Rentrak developed its own set-top box enterprise, which has attracted many TV station clients.  And set-top-box measurement was given as one of the main rationales for creating The Coalition for Innovative Media Measurement (CIMM).

I don’t want to rain on anyone’s parade, but I don’t think we are going to get true “currency” measurement from set-top boxes for a long time -- if ever. Set-top boxes can be a valuable analytical tool, but most of the barriers to set-top-box currency ratings that existed five years ago still remain and don’t seem to be coming down very fast. 

The methodological objections to set-top box measurement are well-known:

1) There are no set-top boxes in homes without cable, which effectively disenfranchises many low-income and minority homes; 2) in many homes, the set-top box stays on after the TV has been turned off, so viewing would be counted when no one is watching; 3) there is no way to tell WHO in the house is watching even when the television is turned on (i.e., no demographics); 4) many homes have multiple TVs but only one set-top box, which means viewing in kids’ rooms and master bedrooms would not be captured; 5) most set-top boxes don’t have the capability to report data back to a measurement company (this  problem is especially true in satellite homes, which would require a connection to a phone line to report back data).

There’s also the practical fact that set-top boxes are not designed to measure TV viewing.  That’s not their job, nor is it the main focus of the cable and satellite companies who own them.  With only a limited amount of information that can flow through cables, many cable companies have made delivering more channels a priority over capturing data. So to obtain viewing information, a measurement company has to jury-rig a reporting system that works with some boxes and not with others. 

What this boils down to is that a set-top box sample may be larger, but wouldn’t be more accurate, because it’s not representative of the entire population. By definition, it would only include homes with cable -- and would furthermore only include boxes capable of reporting back to a measurement company.

Having said all that, I am confident Nielsen can develop a hybrid measurement that combines numbers from both its People Meters and a set-top-box sample.  Of course, that would require some fancy modeling to cover the gaps and address the issues discussed above.  (As an aside, I pity the client services people at Nielsen if the company ever adopts a currency that is based on a model that only stat geniuses can understand.  Nielsen is already accused of using a “secret sauce” to massage the People Meter numbers. If Nielsen adopted modeling, it really would be using a “black box” into which raw numbers are poured, and from which a completely new rating emerges.  This model would probably be blamed every time a network had a ratings decline.)

The real question is whether this hybrid measurement would be better.  Theoretically, a bigger sample might provide more stability for the smaller national networks whose ratings tend to bounce around a lot (if your network only averages a rating of 0.2, this number can fluctuate significantly on any given day because the number of People Meter homes that tune in that network is pretty small.) Then too, set-top boxes could provide some kind of measurement for so-called “long-tail” networks audiences so small they don’t even generate a 0.1 rating.  

But unmeasured networks only constitute about 6% of total viewing, and you have to wonder if the other national networks would be better off with the significant change that a hybrid measurement would entail.  Maybe I drank the Kool-Aid when I worked at Nielsen, but I think ratings for national networks are more accurate when they are drawn from a good, well-maintained panel than from an unrepresentative, albeit larger, sample.

For all these reasons, Nielsen and its largest clients generally agree that set-top data can be useful for analytical purposes, but that the People Meter sample will remain the primary tool for collecting ratings on a national basis.  As a result, little, if any effort is being put into integrating set-top boxes into a national ratings system.

The situation is different at the local level, where paper diaries are still being used to measure local TV stations.  In my next column, I will look at efforts to use set-top-box data in local markets.