Senior executives from Yahoo, MSN, comScore and Nielsen//NetRatings made those points and others during a free-wheeling panel Wednesday on the state of online metrics at the annual meeting of the Advertising Research Foundation.

"We're standing in quicksand here," said Lynn Bolger, vice president of advertising and sales insights for Yahoo, referring to the ever-moving landscape. "It makes it difficult to pick a time and place to develop a process that's meaningful going forward."

"The Internet is changing so fast that if to get accredited you have to lock in one approach, you won't be able to measure the next thing without being unaccredited," said Gian Fulgoni, chairman and co-founder, comScore Networks. "The audit process has to get more flexible."

Interactive Advertising Bureau CEO Randall Rothenberg recently sent an open letter to comScore and Nielsen//NetRatings deriding the use of panels to measure online media and calling for a long-delayed Media Rating Council audit. That has resulted in several cross-industry meetings and opened a dialogue about the discrepancies between server-based audience counts and those reported by the research firms. Both are actively seeking accreditation.

Both Fulgoni (basking a bit in the glow of a successful comScore IPO pricing) and Mainak Mazumdar, chief of measurement science for Nielsen//NetRatings, defended the use of panels as a legitimate tool for new media--and perhaps the only way to get at the critical insight of just how online exposure affects offline behavior.

Cookie deletion is just one factor leading to large discrepancies between server-derived and panel-based counts of unique visitors, Fulgoni noted. For sites that need to offer audience counts to prospective advertisers, comScore is offering ad impressions delivered as a metric of real meaning, he said.

"Emerging media are being held to a higher standard than traditional media," Fulgoni said.

"When you evaluate a campaign, it's more than currency. That's almost like a short-term measure," said Beth Uyenco, global research director, Microsoft Digital Advertising Solutions.

"If you had a 20 share rating you'd want to talk about it," quipped Bolger. She said it will take "gigantic" panels for the media industry to get the kind of information it's had in the past.

If nothing else, research companies should work toward transparency, said Uyenco. "As a data user, if I know how the sausages are made, I can adjust my thinking. I encourage auditing. I want to know what I'm working with and then I can make a decision."

Bolger urged the ARF to help come up with different ways to approach the measurement of online audiences beyond mere counting, and to make uncertainty part of the norm. "We need to prepare the industry for the chaos" of the future, she said.

Mazumdar, pointing out that servers also count "machine-driven" visits from spiders and other BOTS, noted that a tension often exists between the search and measurement sides of a company. Effective optimization and frequent site updating lead to more machine-generated traffic.

"Someone has to get hold of the issue and wrestle it to the ground," said Fulgoni. "Otherwise we'll be here ten years from now debating does a site have 10 million unique users or five?"

This news story is important. I recommend that everyone read it.