| |||||||||||
PRINT
COMMENT
SUBSCRIBE
RSS(DISCLAIMER: As the chief research officer at comScore, I have an agenda in online metrics.)
The Internet, we have been told for a decade, is the most measurable medium. Hence it is not without vexation that the Internet business community observes the current state of online metrics. Because unlike a traditional medium such as network TV, where a single metric system has held uncontested sway for over 50 years, the Internet community is "blessed" with an embarrassment of measurement riches.
We've got comScore and NetRatings, who each collect detailed behavioral and demographic data from a panel of Internet users and then project to the online universe. Among all the online metrics providers, these two companies hew closest to the traditional construct of syndicated audience measurement.
Then there are companies like Alexa, which collects information from Web users who have downloaded and installed their toolbar, and Hitwise, which collects data from ISPs. There are new entrants like Quantcast and Compete, both of whom offer spiffy free online interfaces that allow users to profile a Web site's audience. Quantcast combines data from advertisers, publishers, ISPs, and ad networks; Compete combines data from ISPs, toolbars, and panelists.
Then we've got the Web analytics providers, including companies like Google Analytics, Omniture, WebSideStory, WebTrends, and Unica. These are the sources of what we often call "site-centric data." It is probably worth pointing out that in the Internet metrics space, we really have two distinctly different but overlapping disciplines: audience measurement and Web analytics. Each of these disciplines has its own history, culture, set of objectives, and set of best practices.
Meet Josh Chasin at OMMA Metrics Measurement NYC!
Josh Chasin will be there speaking during "Course Collision" on February 24 at 9:45 AM. Top executives will be there. Will you?
Register today and save.
Register today and save.
The Interactive Advertising Bureau made headlines earlier this year when it called attention to the differences between site-centric Unique Visitor (UV) counts and panel-based UV projections. The implication appeared to be that site-centric counts were a priori correct, while panel-based data was therefore inherently flawed. We refuted this contention in a white paper on the impact of cookie deletion on site-centric UV counts (). We've also observed that there can be issues involving non-human traffic, international traffic, and duplication between work and home usage that can inflate site-centric UV counts.
Lately I've been a member of the Web Analytics Association's listserv, a Yahoo forum founded by Eric Peterson, which is currently about 3,800 members strong. (If you want to learn a lot about Web analytics fast, check it out.) There I've come to learn that a Web publisher can install and run two separate analytics packages simultaneously and end up generating two divergent sets of site-centric data for the same site at the same time (even though both are based on the same so-called "census" of visitor data). In the 2007 Web Analytics Interim Report published this past May, Jim Sterne wrote by way of introduction: "Every web analytics tool measures clickthroughs and page views a little differently. They're all using slightly different yardsticks and getting slightly different results. The disparity is driving us to distraction."
Distraction. Vexation. Same difference.
How might we sort all this divergence and disparity out? Right now, the Media Rating Council (MRC) accredits ad impression counts from AOL, Atlas, CNET, Disney, DoubleClick, MSN, Univision, Weather.com, and Yahoo. Several other publisher audits are in process (http://www.mediaratingcouncil.org/Accredited%20Services.htm). With the exception of Univision, though, the UV counts provided by these entities are not accredited (Univision was granted an exception.) With respect to audience measurement services, currently both comScore and NetRatings are working through the MRC audit process. NetRatings is probably 6 to 8 months further along than we are (My position: it isn't a race.)
But it is unrealistic to expect that widespread MRC auditing will ameliorate the differences in metrics as reported by different sources. Accreditation will not force panel-based metrics and server-centric data into any kind of alignment. This is not to minimize the importance of an MRC audit. The process is extremely valuable for vetting methods and procedures, for assuring transparency and disclosure, and for providing industry constituencies with a window into the measurement companies' methodologies via MRC membership and participation. I have 27 years of experience in syndicated audience measurement and can attest that the audit and accreditation process makes the measurement company better. But it does not force convergent validity when there are multiple measurers.
Well then, maybe what we need is standardization.
Right now the IAB and the MRC are working together to develop a set of IAB Reach Measurement Guidelines (e.g., what is a "Unique?"). Meanwhile, the Web Analytics Association has just published a set of Web Analytics Definitions, which overlaps with the scope of the IAB-led initiative. So maybe we need to standardize the standards?
If the Internet is so darned measurable, why are there so many conflicting measures?
Let's take a breath.
See, here's the thing. One of the consequences of being the most measurable medium is that the Internet ends up as the medium with the most measures. We tend to experience this as paradox-- the Internet is inherently measurable, yet measures are abundant and widely divergent. When you think about it, though, this is not paradoxical at all. Measurability begets measures. We have to stop letting this abundance of metrics keep us from doing the business we need to do.
It is my hope and intention that this forum -- the Online Metrics Insider, and my own humble contributions hereto -- will help serve as a platform for placing the metrics in context, sorting them out, getting past the paralysis that multiple metrics sources can sometimes induce.
I'm thrilled to be working in this space at this time -- in the medium with the most measures. Let's see what we can do together to take advantage of this embarrassment of riches. Because as far as media measurement is concerned, the current online metrics environment will prove to be the rule, not the exception. Already, more than half of all U.S. ad impressions are served digitally, and that ratio will only increase. Digital media platforms -- the Internet, digital cable TV, the fast-emerging wireless-cellular medium -- will continue to beget and support multiple metrics sources. Alignment around a single metric source should not be perceived as a step toward legitimacy for the online medium; it should be perceived as a step backward, back into the 20th century, back to the ways of analog measurement.
I think we're ready to move forward. In fact, I'm counting on it.
1 person recommends this article.
RSS FEEDS
Mathematically, the solution to the problem of equating different metrics was provided almost 50 years ago by Georg Rasch, a Danish mathematician who had studied with Ronald Fisher in London in the 1930s. Rasch's models have been used in high stakes admissions and professional certification testing for decades. They are valued for their legal and scientific defensibility.
The situation in online metrics described by Josh is analogous to the kind of data generated by computer adaptive tests. With a calibrated bank of test or survey items, each examinee or respondent can be administered a unique set of items without compromising the comparability of the metric. These different sets of items might all belong to one owner (vendor, metric provider, etc.) or they might have been brought together only for the purposes of equating, so that they can be made to all report out in the same additive, invariant currency of exchange.
Just about anything you would ever need in the way of pointers toward more information in this area is available at http://www.rasch.org. Rasch Measurement Transactions is a full text quarterly bulletin available at that address, now going into its 21st year. The second edition of the best-selling book on fundamental measurement by Bond and Fox just came out this past Spring (see below for citation and URL).
Several software packages are available; a near-comprehensive directory is at http://www.winsteps.com/rasch.htm. For applications in the marketing and business domains, see the references below.
A good introduction to the basic issues relative to marketing research is provided by Thomas Salzberger at http://smib.vuw.ac.nz:8081/www/ANZMAC1999/Site/S/Salzberger.pdf. Salzberger's publications list is at http://www2.wu-wien.ac.at/marketing/redsys/paragraphs.php3?article_id=109, though you'll have to read German to access most of it.
Allerup P, Bech P, Loldrup D, Alvarez P, Banegil T, Styles I et al. Psychiatric, business, and psychological applications of fundamental measurement models. International Journal of Educational Research 1994;21(6):611-22.
Bechtel GG. Generalizing the Rasch Model for consumer rating scales. Marketing Science 1985 Winter;4(1):62-73.
Bechtel GG, Wiley JB. Probabilistic measurement of attributes: A logit analysis by generalized least squares. Marketing Science 1983 Autumn;2(4):389-405.
Bond T, Fox C. Applying the Rasch model: fundamental measurement in the human sciences, 2d edition. Mahwah, New Jersey: Lawrence Erlbaum Associates [http://homes.jcu.edu.au/~edtgb/book/], 2007.
Dodd BG, De Ayala R, Koch WR. Computerized adaptive testing with polytomous items. Applied Psychological Measurement 1995 Mar;19(1):5-22.
Fisher WP Jr. Living capital metrics. Rasch Measurement Transactions 2007;21(1):1092-3 [http://www.rasch.org/rmt/rmt211.pdf].
Lunz ME, Bergstrom BA, Gershon RC. Computer adaptive testing. International Journal of Educational Research 1994;21(6):623-34.
Lunz ME, Linacre JM. Measurement designs using multifacet Rasch modeling. In: Marcoulides GA, editor. Modern methods for business research. Methodology for business and management. Mahwah, New Jersey: Lawrence Erlbaum Associates, Inc., 1998. p 47-77.
Salzberger T, Sinkovics RR. Reconsidering the problem of data equivalence in international marketing research: Contrasting approaches based on CFA and the Rasch model for measurement. International Marketing Review 2006;23(4):390-417.
Singh J, Rhoads GK, Howell RD. Adapting marketing surveys to individual respondents. Journal of the Market Research Society 1992 Apr;34(2):125-47.
Thanks.
PS: Lisa Barone at www.Bruceclay.com did a really nice recap of SES San Jose 2007 in a post titled "Issues In Analytics", where she highlights the key issues discussed during a panel discussion about why the numbers don’t match.
Right now, when we say the internet is the most measurable medium, we simply mean that internet measurement produces the most information. That wealth of information should lead to better measurement, but initiatives that will do this are only recently moving forward, through the MRC and IAB on the audience front. The opportunity is to have the most meaningful and relevant measurement of any medium.
For example, I've found that since you can better track profitability or lack thereof, individual online marketing efforts are terminated sooner than traditional ones...
And yet it's still impossible to track everything... web views to retail buys, for example. Not everyone cuts coupons. Yuck!