• by Frank S. Foster , March 17, 2008

I had a very interesting conversation the other day with an industry veteran regarding the state of the television audience measurement industry. As someone who has been a spokesperson for new ways of measuring television, I am occasionally approached by people who assume that I have no idea how it has always been done. The conversation began innocently with concerns about set-top-box data, each of which I answered simply and directly.

After 30 minutes, he had nothing more to say. When I suggested counting one in five anonymous households and mathematically inferring demographic attributes would be a more acceptable solution than projecting from a small panel, he became quite agitated. "The panel is the only acceptable solution," he cried. "No other approach allows us to taste the soup!"

The infamous vegetable soup analogy had been thrown down. Now, this particular gentleman should have known better -- he was granted an advanced degree from a prestigious college -- because the analogy borders on the ridiculous. But when faced with an argument he was not winning, he turned to something familiar. I told him I was running late but that I would address the issue in my next MediaPost TV Board column. So here goes.

I googled "Nielsen Media Research Soup" and found a white paper of sorts with Nielsen's logo entitled "What TV Ratings Really Mean." Below is an excerpt:

"Actually, a representative sample doesn't have to be very large to represent the population it is drawn from. For example, you don't need to eat an entire pot of vegetable soup to know what kind of soup it is. A cup of soup is more than adequate to represent what is in the pot. If, however, you don't stir the soup to make sure that all of the various ingredients have a good chance of ending up in the cup, you might just pour a cup of vegetable broth. Stirring the soup is a way to make sure that the sample you draw represents all the different parts of what is in the pot."

"A cup of soup is more than adequate to represent what is in the pot." That seems a fair assumption, but the application to television audience measurement is problematic. I believe Nielsen projected that there will be 112,800,000 television households in the U.S. market for the 2007-2008 television season. I believe it is generally accepted that the effective size of the current national people meter panel is 10,000, although many think it will grow to 17,000 by 2011.

To make my point, I will be generous and assume the effective sample will be 20,000 at some point, and that the number of television households will remain constant. 20,000 out of 112,800,000 equates to 1 in 5,640 households. So how small does that make Nielsen's cup?

When I was a child, the biggest pot in my mother's home was a five-gallon soup kitchen pot. There are 16 cups to a gallon, so there must have been 80 cups in her soup kitchen pot. So Nielsen's "cup" isn't really a cup. There are 256 tablespoons in a gallon, so there must have been 1,280 tablespoons in my mother's soup kitchen pot. So Nielsen's "cup" isn't really a tablespoon. If there are 768 teaspoons in a gallon, there must have been 3,840 teaspoons of soup in the soup kitchen pot. But that means a teaspoon is still too big to use and Nielsen's "cup" is smaller than a teaspoon. According to Wikipedia, there are 60 "drops" in a teaspoon, so there must have been 230,400 drops in my mother's soup kitchen pot. Which means Nielsen's "cup" actually translates to 41 drops -- or approximately 2/3 of a teaspoon.

What further complicates the analogy is that Nielsen's one pot is used to measure television in its entirety -- which consists not of one network, but hundreds. How do you measure hundreds of networks with just 41 drops? Assume a broadcast network had a great night and 20% of the nation's population watched that network's program. Nielsen's "cup" for that network would be 20% of 41 drops, or approximately eight drops. Out of those eight drops Nielsen would have to differentiate by age, sex and race. A popular network with a 2.0 rating would have one drop of soup, give or take. The ratings for the vast majority of networks would be based on much less soup than that.

Keep in mind we have not discussed the "stirring" of the sample, nor have we talked about those cups that are drawn from the pot and dumped because they would not agree to take part in the tasting. We have also limited our analysis to Nielsen's national panel. Their local panels are an order of magnitude smaller and more problematic. If we include geographic targeting (Sys Code Clusters, for example) the issue becomes further complicated. Of course, addressable advertising cannot be measured by a sample at all.

To my learned colleague with whom I spoke the other day, this has been a real-world examination of the vegetable soup analogy. The small sample approach was fine when collecting more data was expensive and wrought with problems, but it makes little sense today. Ultimately, it will be the television industry that will decide if small panels remain the "gold standard." I continue to believe counting one in five households and using mathematical inference to determine demographics is a vastly superior solution.

Undoubtedly, there will be those who disagree. Rebuttals can be posted on this blog or addressed directly to me. My contact information is listed below.