Calling ALL Internet Users
by Josh Chasin,
Jan 20, 2009, 10:30 AM
I grew up in the field of audience measurement. I was working in the Statistical Services department at Arbitron in 1980, before I was out of college; I would take the subway uptown from Washington Square wearing a shirt and tie under my leather jacket. In fact I spent the first seven years of my professional career in that department, working with and learning from some of the all-stars and leading lights of media research. I remember making a suggestion one time, and my boss Mike Occhiogrosso smiling at me, impressed, and saying, "We'll make a sampling statistician out of you yet." While at Arbitron I even had a chance to work with Hugh Malcolm Beville, the first executive director of what is now the MRC, and the man who, literally, wrote the book on audience measurement.
I'm indulging in this little trip down memory lane to make the point that I had a pretty good bottom-up schooling in the discipline of audience measurement; a practical education. I want to make this point, because over the past five years I've had to unlearn everything I knew about audience measurement. That may be overly dramatic, but I have come to view the discipline I grew up with as "analog" audience measurement, whereas what we need to deploy today is digital, or 21st century, audience measurement (I'm resisting the temptation to call it "audience measurement 2.0.") Dr. Cheryl Harris and I presented a paper at the ARF's first Audience Measurement Symposium in 2006, titled "The Effect of Technological Innovation on Media Exposure Tracking: In Search of the New Traditional." Our hypothesis was that changes in the technological landscape were rendering traditional measurement methods and techniques obsolete, but that solutions -- the "new traditional" -- would likewise emerge from the application of technology.
This paper pre-dated my association with comScore. But it helped convince the folks at comScore that I was the guy for them.
One of the bedrock assumptions of the analog era of audience measurement was that high quality audience measurement began with a Random Digit Dial (RDD) frame. Sure, 2% or so of American households lacked phone service, but that was a bias we were all prepared to live with. And back when I started at Arbitron, if you were doing a telephone survey and you weren't getting a response rate of 70% or more, you were doing something wrong. So RDD, with near-complete universe coverage in the frame, and robust response rates, was the path to projectable and representative studies.
We all know what's happened to survey response rates since then. And we know, too, that the cell phone is rapidly making the landline, the "house phone," a thing of the past. (One change as we become a cell phone nation is that the telephone number is increasingly associated with a person, not a household; if you want to reach me you call my cell, and if you want my wife you call hers. Using telephones to sample households as opposed to persons will become increasingly difficult.)
The CDC's National Health Interview Survey has become a highly visible source in the research community for tracking the incidence of what they call wireless-only, and what media researchers generally call cell phone-only, households. On Dec. 17, their latest findings were released, covering the first six months of 2008. According to the NHIS, 17.5% of US households were wireless only. To put that into perspective, just three years prior, in the same study, only 7.3% of US households were wireless-only.
When we look at target demographics, the cell-only situation becomes even more dramatic. Fully 21.6% of US Hispanics live in cell-only households. And consider this: 31.4% of 18-24 year-olds live in cell-only households, as do 35.7%-- over a third!-of 25-29 year-olds. Now, this might be a function of age and life stage, but it might also be a generational phenomenon. In other words, I don't expect all those 18-29 year-olds to wake up one day, say, "Hey, I'm 40, I have a house in the suburbs-time to get a land line!" No. These kids are as likely to get a landline as they are to start buying vinyl LPs.
Why is this a research issue? Well, most RDD is done at calling centers, using auto-dial systems that automatically place hundreds of calls, handing off the call to a live interviewer when someone answers. But it is against the law to auto-dial cell phones. So RDD sampling systematically and by design excludes cell phone exchanges. In order to sample cell phones, a human has to manually dial the number, rendering the process several times more costly than RDD dialing of land lines.
In May of 2008, the Media Rating Council (MRC) issued a statement that "Accreditation will be at risk for any service affected by this issue that does not have a known solution and timeline approved by MRC to effectively measure cell phone only households."
But I think the issue is even more complex than cell-only. Because there are more than two flavors of households (those with landlines, and those with cell phones only.) In their recent release, the NHIS reports that another 13.3% of US households have landlines, but "received all or almost all their calls" via cell phone. At comScore we've been using the term "cell phone primary" to describe this type of household.
ComScore moved away from the "analog age" recruitment technique of RDD some time ago, and we recruit our panelists exclusively online (save for our "Calibration sample," a control sample that is recruited randomly and offline, but not via RDD.). This has many benefits, but for today's purposes, it allows us to assure that we represent persons from all phone-status households. A year ago we did a study of our U.S. panel, in order to understand both phone status composition of the panel, and how Web usage might vary by phone status. We found that 19% of our panelists were cell-only, and that another 23% were cell-primary; in other words, if we'd relied on RDD we would have totally excluded 19% of the panel, and dramatically under-represented another 23%.
More important, we found that Internet usage behavior varied markedly based on phone status. Cell-only persons were heavier users of rich media entertainment content, games, and conversational media; cell-primary were heavier users of travel, business/finance, and other service-oriented content.
The macro lesson here is clear: that we can no longer expect representative samples in media research to accrue from RDD. The bigger lesson, and perhaps the more difficult one to assimilate, is that we as an industry need to revisit all the bedrock assumptions we grew up with -- especially those of you as old as me.
I'm indulging in this little trip down memory lane to make the point that I had a pretty good bottom-up schooling in the discipline of audience measurement; a practical education. I want to make this point, because over the past five years I've had to unlearn everything I knew about audience measurement. That may be overly dramatic, but I have come to view the discipline I grew up with as "analog" audience measurement, whereas what we need to deploy today is digital, or 21st century, audience measurement (I'm resisting the temptation to call it "audience measurement 2.0.") Dr. Cheryl Harris and I presented a paper at the ARF's first Audience Measurement Symposium in 2006, titled "The Effect of Technological Innovation on Media Exposure Tracking: In Search of the New Traditional." Our hypothesis was that changes in the technological landscape were rendering traditional measurement methods and techniques obsolete, but that solutions -- the "new traditional" -- would likewise emerge from the application of technology.
This paper pre-dated my association with comScore. But it helped convince the folks at comScore that I was the guy for them.
One of the bedrock assumptions of the analog era of audience measurement was that high quality audience measurement began with a Random Digit Dial (RDD) frame. Sure, 2% or so of American households lacked phone service, but that was a bias we were all prepared to live with. And back when I started at Arbitron, if you were doing a telephone survey and you weren't getting a response rate of 70% or more, you were doing something wrong. So RDD, with near-complete universe coverage in the frame, and robust response rates, was the path to projectable and representative studies.
We all know what's happened to survey response rates since then. And we know, too, that the cell phone is rapidly making the landline, the "house phone," a thing of the past. (One change as we become a cell phone nation is that the telephone number is increasingly associated with a person, not a household; if you want to reach me you call my cell, and if you want my wife you call hers. Using telephones to sample households as opposed to persons will become increasingly difficult.)
The CDC's National Health Interview Survey has become a highly visible source in the research community for tracking the incidence of what they call wireless-only, and what media researchers generally call cell phone-only, households. On Dec. 17, their latest findings were released, covering the first six months of 2008. According to the NHIS, 17.5% of US households were wireless only. To put that into perspective, just three years prior, in the same study, only 7.3% of US households were wireless-only.
When we look at target demographics, the cell-only situation becomes even more dramatic. Fully 21.6% of US Hispanics live in cell-only households. And consider this: 31.4% of 18-24 year-olds live in cell-only households, as do 35.7%-- over a third!-of 25-29 year-olds. Now, this might be a function of age and life stage, but it might also be a generational phenomenon. In other words, I don't expect all those 18-29 year-olds to wake up one day, say, "Hey, I'm 40, I have a house in the suburbs-time to get a land line!" No. These kids are as likely to get a landline as they are to start buying vinyl LPs.
Why is this a research issue? Well, most RDD is done at calling centers, using auto-dial systems that automatically place hundreds of calls, handing off the call to a live interviewer when someone answers. But it is against the law to auto-dial cell phones. So RDD sampling systematically and by design excludes cell phone exchanges. In order to sample cell phones, a human has to manually dial the number, rendering the process several times more costly than RDD dialing of land lines.
In May of 2008, the Media Rating Council (MRC) issued a statement that "Accreditation will be at risk for any service affected by this issue that does not have a known solution and timeline approved by MRC to effectively measure cell phone only households."
But I think the issue is even more complex than cell-only. Because there are more than two flavors of households (those with landlines, and those with cell phones only.) In their recent release, the NHIS reports that another 13.3% of US households have landlines, but "received all or almost all their calls" via cell phone. At comScore we've been using the term "cell phone primary" to describe this type of household.
ComScore moved away from the "analog age" recruitment technique of RDD some time ago, and we recruit our panelists exclusively online (save for our "Calibration sample," a control sample that is recruited randomly and offline, but not via RDD.). This has many benefits, but for today's purposes, it allows us to assure that we represent persons from all phone-status households. A year ago we did a study of our U.S. panel, in order to understand both phone status composition of the panel, and how Web usage might vary by phone status. We found that 19% of our panelists were cell-only, and that another 23% were cell-primary; in other words, if we'd relied on RDD we would have totally excluded 19% of the panel, and dramatically under-represented another 23%.
More important, we found that Internet usage behavior varied markedly based on phone status. Cell-only persons were heavier users of rich media entertainment content, games, and conversational media; cell-primary were heavier users of travel, business/finance, and other service-oriented content.
The macro lesson here is clear: that we can no longer expect representative samples in media research to accrue from RDD. The bigger lesson, and perhaps the more difficult one to assimilate, is that we as an industry need to revisit all the bedrock assumptions we grew up with -- especially those of you as old as me.
0 comments on "Calling ALL Internet Users".
Leave a Comment
Sign in to leave a comment. Don't have an account? Join Now
Recent Metrics Insider Articles
-
Viewability And RTB: Notes On The Larger Context May 21, 10:24 p.m.
In a May 8 post, Alex White makes some good points about how viewability measurement will ...
-
Attribute That! May 14, 1:10 p.m.
Attribution modeling or path-to-purchase analysis? These concepts are often used in the same context -- and, ...
-
Bring On Good Measurement! May 8, 9:31 a.m.
Online advertisers are blinding themselves. And they’re doing it on purpose. The digital channel enables us ...
-
Better Safe Than Sorry May 2, 1:12 a.m.
After months of writing and speaking about Making Measurement Makes Sense (3MS), on my own and ...
-
Industry Trend: Higher Expectations As Marketing Attribution Matures April 25, 2:41 p.m.
As with any service, technology, or combination of both, the expectations of the marketplace grow more ...
-
Out Of Chaos, The Path To Purchase April 17, 7:13 a.m.
There is one diagram that any marketer would be capable of sketching from memory, even after ...
-
Programmatic Buying Meets Attributed Metrics: A Match Made With Big Data March 26, 4:26 p.m.
One of the latest trends in today’s digital marketing ecosystem involves the intersection of Big Data, ...
-
Data Scientists Swim, Surf, Pick And Juggle March 15, 6:11 p.m.
As mobile advertising specialists, we depend on the work of our data scientists. They’re the ones ...
-
How To Become A Data-Centric Organization March 5, 12:02 p.m.
Everywhere you turn these days, there’s an article, conference or new tech solution about Big Data. ...
-
Silence Is Golden Feb. 21, 11:03 a.m.
It seems the notion that silence is golden has lost its luster. As an industry, a ...
As a research geek myself, this is moderately interesting stuff, and it's interesting to see that the US is catching up with Europe in terms of wireless phone incidence and moving away from landline-only sampling.
But really, is it right that this MediaPost column should be a pulpit to tout the performance of comScore at conducting research in such and such a way? If I want to know how comScore do their sampling in my region, I'll ask my comScore customer service rep. If I want to read a column about broad implications of measurement techniques as they apply in general across the industry... well, I won't read a MediaPost column about it, clearly!
Given your expertise and experience in traditional measurement I wonder where you think things are going with observational research tools like SM2, our social media monitoring and analytics solution? More akin to a kind of super mystery shopping methodology, social media monitoring collects brand conversations that are unprejudiced by the need to select panelists or write surveys. We feel that this kind of data (we collect everything in social media then provide tools for searching, understanding and responding to it) will change the face of market research.
Nick:
Point taken. Wen I first began writing these columns for Mediapost (and really, they are somewhere between columns and contributions to a blog), I wrestled with proper approach to voice, and even talked about it with the Mediapost editors. THe approach I've chosen to take, and one which my editors at Mediapost have encouraged, is to write from the perspective I do in fact have, and to make that perspective (my role at comScore) explicitly clear. Instead of ignoring the fact that I'm the comScore guy, I endeavor to embrace iit, and to indulge in the conceit of providing a window into what I deal with as comScore Chief Research Officer. But I will continue to rely on readers such as yourself to keep me honest!
Thanks.
--josh--