• by Gary Holmes , Op-Ed Contributor, October 10, 2016

The recent news that Nielsen intends to eliminate paper diaries from the 140 ratings markets in which they are being used was met with relief in some quarters — along with incredulity in others that such antediluvian methods were still being deployed to generate TV ratings in the 21st century.

For me, however, it brought back memories from more than a decade ago, when Nielsen first announced its intention to phase paper diaries out by 2011.

Paper diaries harken back to the dawn of the television age, when there were only a handful of TV networks and viewers could generally be counted on to remember what channels they watched throughout the day. But even then, Nielsen founder Art Nielsen was searching for a better electric measurement system that would record what people actually watched instead of what they said they watched.

Fast-forward to June 2006. Coming off a bruising fight with News Corp over the introduction of the Local People Meters that replaced these same paper diaries in the top 10 local markets, Nielsen faced a more implacable foe: the Internet.  People were beginning to find new ways to watch TV online and Nielsen needed a plan to adapt.

The result was an initiative called Anywhere Anytime Media Measurement, aka A2M2. Looking back on that plan 10 years later is a reminder of how visionary corporate aspirations are often restrained by more mundane considerations, such as cost, technology and clients.  

For example, it turned out that clients weren’t interested in paying Nielsen to measure viewing outside the home (although that is apparently back on the table again.)  And when it was clear iPods wouldn’t become the primary vehicle for mobile viewing, Nielsen dropped the quest to develop a “go meter.” 

On the plus side, Nielsen did eventually integrate Internet viewing into its ratings, and it did expand the number of local markets measured with People Meters.

The nut it couldn’t crack was the complete elimination of diaries.  Nielsen’s ratings CEO Susan Whiting had pushed for a deadline of introducing electronic measurement in all local markets by 2011, reasoning that the Nielsen staff would be motivated into action by an aggressive but firm deadline. 

They considered and tested a number of options, including a “mailable meter” that would be sent to viewers in lieu of a diary and placed next to a TV for a month, where it would record what shows were being watched, and then returned to Nielsen for transcription.

A lot of work went into developing and testing this mailable meter, but in the end, it wasn’t good or cheap enough to replace paper diaries. Here’s the thing about diaries: They are cheap. Nielsen lost money measuring many of these markets, but it lost less than it would have with electronic measurement.

Here’s the other thing about diaries: They hung around so long because a lot of the people who pay the bills (that is, the local stations) didn’t really want to change if it meant lower ratings.  TV viewers are most likely to mark down the big-name shows they “usually” watch, which are typically network shows — which inflates network ratings.

So there was an incentive to maintain the status quo.  

Or at least, there used to be. In a world of streaming, video playback, on-demand, channel-switching, premium cable stations, and limited attention spans, viewers are less likely to remember anything they watch, even the six o’clock news, and the diaries have finally lost any credibility among advertisers.  The incentive for abandoning diaries is finally bigger than it is for keeping them.

What’s coming instead is something that’s been in Nielsen’s toolbox for years: fusing together different datasets, including return-path data from set-top boxes and data from other electronic sources like the National People Meter sample.   The flaws of set-top box data are numerous, including a lack of representation and no information on which person in the house is watching, but at this point anything would be better than paper diaries.

The adoption of fusion data based on modeling different datasets is a significant development. 

Until now, almost all TV ratings have been based on quantifiable data from scientifically selected panels that can be double-checked.  You can go back and see how many people actually pushed People Meter buttons in a market or wrote in particular programs in their diaries. 

There’s even a room at Nielsen’s Oldsmar facility where station managers can go to review the individual diaries from their markets and confirm that, yes, this 53-year-old white woman really did watch “The Wheel of Fortune” instead of “Jeopardy.”  You can’t do that when data from set-top boxes are funneled into a computer, fused with other data, and modeled using an algorithm that only a handful of data scientists can understand.

Welcome to the 21st century. I’m sure my former Nielsen colleagues are thrilled they’ve finally convinced the market to phase out diaries, which symbolized the old, analog Nielsen in a digital age.  That’s an image no one wanted.

The new target date for eliminating diaries is now 2018, seven years after the first deadline.  But I’d bet on Nielsen making this one.  Not only is the technology there — but so is the marketplace.