Nielsen Glitch Focuses Attention On New Online Measurement System, Impact Unknown

A transition that would likely have sparked a firestorm of debate in Nielsen's core medium of television - a complete overhaul of the system it uses to produce its audience estimates - has gone virtually unnoticed in the online industry. That was until a few weeks ago, when a routine investigation revealed a strange and unexpected glitch in the methods Nielsen currently uses that grossly exaggerated the online audiences of three of the Web's best-known publishers: Fox News, The New York Times, and USA Today (see related story). Nielsen today will release corrected, albeit lower online audience estimates for the month of May for those publishers, and it says the new system will prevent problems like that from ever happening again, but the gaff is finally getting people in the online business to focus on the impact a new Nielsen audience ratings system will have for Web publishers, advertisers and agencies alike.

While the new system will clearly be an improvement by many standards - including a new sample that will be 10 times larger than the one it currently uses to measure online audiences, enabling Nielsen to dramatically increase the number of Web sites it reports data for - a change of that magnitude also is likely to alter the relative shares of online publishers that depend on Nielsen's data.

It's a methodological change, one long-time Nielsen client says, "could rival the people meter change of the late 1980s." The client was referring to the TV industry's transition to people meters in 1987 from the older, arguably less accurate meter and paper diary system Nielsen used before. While a methodological improvement, that change sent ratings for the major broadcast networks plummeting, and cost them millions of dollars in lost advertising revenues. Even in recently, years after years of vetting, the transition to people meters in local TV markets has caused TV industry eruptions, including a federal antitrust suit filed against Nielsen by Sunbeam Television Corp.

Nielsen executives say it is routine for clients to "shoot the messenger," or blame Nielsen when it changes its methodology, even when it's an improvement, if the changes reduce their audience estimates and decrease their share of advertising dollars. And while Nielsen's Web audience estimates don't hold the same level of "currency" value in the online industry that they do in television, some big Web publishers have been conducting their own investigations into Nielsen's methods - though they have been oddly silent about the imminent change.

The New York Times has been most vocal in recent months, insisting that the downward trend in its Web audience numbers are a result of Nielsen's methods and do not reflect a genuine shift in online user behavior.

"We believe their data is in error," New York Times spokeswoman Diane McNulty said. "We're reviewing the methodology now as the numbers are inconsistent with three other independent sources and our own internal data."

New York Times executives declined to elaborate, but McNulty's comment refers to the fact that unlike television, the online industry has a multitude of sources for estimating, and presumably validating, online audience counts. But even in the online world, the Nielsen name carries a lot of water, especially as Nielsen moves to integrate its online, mobile, TV, and "other screen" measurement systems in a way that other research companies may not be able to.

In fact, Nielsen Online's new system, dubbed NetView RDD//Online, will further integrate its online and TV audience measurement systems. Three thousand of the households in the new online measurement system's 200,000 households sample come directly from Nielsen's TV audience measurement system - a soc-called "convergence panel" that is a subset of its national TV ratings service that measures both TV and online usage behavior.

On the plus side, the new online measurement system will be ten times the size of its old NetView system, which represents the last remnants of the original NetRatings measurement system based on the so-called "PC meter" technology that Nielsen entered the online industry with nearly a decade ago. The expansion of the NetView sample may have another unintended consequence for big Web publishers that currently dominate the online display advertising marketplace. By moving from a sample of 20,000 to a sample of 200,000, NetView will be able to expand the number of sites it measures and reports data for from about 3,000 currently to 25,000 with the new system. While that is clearly good for Nielsen's business, the expansion will effectively fragment and dilute the online advertising marketplace, enabling many more, and much smaller Web publishers to compete with the industry's biggest, and to tout the same Nielsen-branded audience estimates that companies such as the New York Times, News Corp., Gannett and others use.

All this occurs, of course, as the Web publishing industry grapples with an economic downturn, as well as what some see as a "secular" shift in the online display advertising business (see related story) that may be commoditizing online advertising prices, even as companies such as the ones mentioned above are trying to wean themselves off higher-valued analog publishing dollars to what are still lower priced digital advertising rates.

But the biggest problem in the shift occurring this summer, is that online publishers, as well as advertisers and agencies, simply will not have enough data to plan, project, and prepare for the shifts that are likely to occur by changing the measurement system. Typically, when Nielsen makes such a wholesale change in its TV measurement systems, it gives the TV industry a year's worth of "parallel" data, running the old and new systems currently for 12-months to allow its clients to prepare for the impact of the change. That happened in 1986-87 when Nielsen converted to people meters, and it happened more recently when it switched from average TV program ratings to time-shifted commercial ratings. Even then, the TV industry has a fair amount of uncertainty and anxiety with the shifts.

But Nielsen is only providing five-months worth of parallel data for its online measurement system shift. Ironically, it had only planned to issue three-months worth of evaluations data before turning the new system on, but opted to extend that another two months when its audience measurement glitch was brought to light. But even that may not be enough time or data to allow the online industry to plan for its impact.

"The biggest fear that everyone has is that they have no way of forecasting what will happen," says one Nielsen client, adding, "While I don't expect the number to be unrecognizable, Nielsen is making wholesale changes - a new meter, a new sample, a new method, and new projection techniques for weighting the results on a month-to-month basis."

Nielsen executives acknowledge that such changes are vexing for an industry that depends on its data to conduct commerce, but they say the trade-off is necessary because of the improvements that will come along with the disruption. In addition to the increase in the sample size, and the ability to measure and report many more Web sites, Nielsen executives say the sample will be far more "projectable" and more representative of the population because it will utilize a better "random probability" method to recruit panelists. The core of that method is called "random digit dialing," a phone recruiting method that has been used successfully in television for years. In addition, Nielsen said it would augment that with "area probability" methods, and would also include "cell phone-only" homes not normally accessed via random digital dialing, and also has means for recruiting "more Hispanic homes," and measuring "secondary computers" within its sample households. In short, the system will be better, more accurate, and more representative than its current system. But it will also be very different, and presumably will yield dramatically different numbers for the online industry.

"As the size of the Internet has grown, our clients wanted a measurement with more breadth and depth, without sacrificing the quality of the measurement," a Nielsen spokesman explained.

One other issue Nielsen will need to deal with is its industry accreditation process. Nielsen, which like its closest rival comScore, has been years into a multi-faceted accreditation process being conducted with media industry ratings watchdog the Media Rating Council, will now have to zero-base that process, and start from scratch.

"They are in the process of accrediting the whole new methodology," the Nielsen spokesman said. "We've had a number of pre-audits that have gone well."

Nielsen//NetRatings NetView

Nielsen//NetRatings NetView

2 comments about "Nielsen Glitch Focuses Attention On New Online Measurement System, Impact Unknown".
Check to receive email when comments are posted.
  1. Mike Kelly from LIN Media, June 12, 2009 at 1:49 p.m.

    How Nielsen can expect to get cred in new media metrics when they continue to measure most TV audiences with paper one week viewing diaries is beyond me.

  2. John Grono from GAP Research, June 13, 2009 at 8:28 a.m.

    There are many facets at play here, and I would like to comment on just two of them.

    First, panel-based measurement of the online world will ALWAYS under-estimate the 'true' audience (if it could be known). This is for the simple fact that a lot of online activity is conducted 'at work', unlike television where viewing is dominated by 'at home'. The reason 'at work' is always under-cooked has little to do with the research company conducting the research but more to do with the fact that the gross majority(90%+) of businesses will not allow metering software on their computers. The businesses that do tend to be small businesses.

    Second, server-based measurement of the online world will ALWAYS overstate the true unique audience. Again, the reason is simple but many faceted. The primary reason is cookie deletion over the period of reporting, but issues such as accessing the internet both at work and at home (doubling the unique audience) along with failing to purge the traffic file of non-human traffic (bots, crawlers, spiders etc), all serve to grossly over-estimate the audience.

    The key question, is how well does each approach work? Well using Australian data (we have a population of 21.7m) face-to-face interviewing and CATI interviewing tend to produce results of between 75%-80% of respondents claim to have "used the Internet in the past month". That is, when we factor out the population under 2 (which we don't measure) and apply this, we're looking at a total 'audience' of somewhere between 16m and 17m.

    OK, how does the panel perform? Well it understates the online population by around 25% with an estimate in the low 12m. As explained, this is because of the "at work" problem, but by any measures the reported data is way too low.

    However, when we look at the server-based unique audience over a month we get over 45m uniques. Yes. The audience is more than twice the total population.

    Yes, the same applies in the US and other countries. Yes, these are the preferred data of all the online sales reps. Yes, these are the data that are held up as facts to tear down the data of research companies like Nielsen and ComScore.

    What I can see is that Nielsen and Comscore are working hard to identify deficencies in their data and to improve them. I wish I could say the same for the online publishers. Neither system is perfect - and no system will be. What we need is a system that uses BOTH measures to produce the most believable audience data.

    As a side note, I think Nielsen would jump at having every TV market metered - but it is simply unaffordable for every market. What that has to do with measuring online media is beyond my fathoming, apart from noting that their data is closer to "the truth" than the oft-touted and much-revered publisher data. The moral of the story is that new media need to look in their own backyard before casting stones ... or are they happy to keep this as their own little secret?

Next story loading loading..