| ||||||||||||
PRINT
COMMENT
SUBSCRIBE
RSSBut what is it anyway that we would hope to capture if we had the right technology and method? In my second article, I suggested that we need a definition of engagement that describes the complexity of audience interaction. We then turned to a model that Eric T. Peterson has been vetting on his blog for over a year. Peterson's approach acknowledges the complexity of engagement by looking at a set of engagement categories that supplement and complement each other.
In the last of this three-part series we will look at using this engagement model in a way that can guide online media strategies. Three questions come into play: how does this engagement model work in practice, who will produce and verify the data, and how can publishers and advertisers collaborate to yield actionable intelligence?
How it works in practice:
In my previous article we defined engagement as the nature of visitors' relationship with a site and how that is expressed in the full range of user interaction, involvement and connection. We then adapted Eric Peterson's model for indexing categories of engagement that effectively describe the kinds of engagement that would illuminate and differentiate Web sites for advertisers: loyalty, recency, click depth, interactivity, duration, and subscription. (Peterson also adds "feedback" and "brand" indices, which are, I believe, not as germane to the outbound advertising sales model). Here is a quick summary of how we can use the engagement index. Each category is indexed according to averages. The index points to the percentage of visitors exceeding the average. If we determine that the average visitor returns to health and medical content sites 4X per year, and 57% of WebMD's audience returns more than 4X per year, the WebMD Loyalty Index would be 57%. Using hypothetical examples of category averages, as we did with WebMD, let's take a look at the indices in action:
- Click Depth (content clicked on): Percentage of visitors who exceed average page views in a given content category. If 26% of visitors exceed, say, 3 page views, C = 26%.
- Loyalty (number of return visits over a longer period of time -- say 12 months): see WebMD example above: L = 57%.
- Recency (number of return visits over a shorter period of time, say 1 month): If 5% of visitors return more than once a month, R = 5%.
- Duration (time of session): If a category of content sites records a 4.6 minute average session time and 19% of a specific site's visitors spend more than 4.6 minutes, then D = 19% for that site.
The next two categories are not indexed against industry averages but are derived from the percentage of users performing specific actions:
- Interactivity (defined actions taken with content-downloading, posting, attending a video or audiocast, etc): If 32% visitors take any one of these actions, I = 32%, during a specified time period.
- Subscription (commitment of name and business or personal info): Measures the percentage of visitors who have given registration information. If 21% of a site's traffic can be identified by name and other submitted information, then S = 21%.
We can then also develop a Total Engagement Index by adding the values for each engagement category and dividing by 6. For the above fictitious example, then, the TE for this site would be 27%.
Would this be a useful way of measuring engagement? Comments are most welcome, of course. As I see it, the Peterson model enables us to acknowledge the complexity of engagement by showing both individual facets of the visitor relationships with a site and a metric for engagement in its entirety. Indexing allows for relational comparisons. We can compare a site to another site or see it in the context of a grouping of sites. And referring to each engagement category, as we noted last week, enables us to see the various dimensions of value that a site holds for its visitors.
If an advertiser knows that the click depth index is fairly low but that the interactivity index is relatively high, one might consider a type of advertising that plays into that strength -- say an audiocast. Or if the subscription rate is high, perhaps that means an opportunity for lead generation activity or newsletter sponsorships. For advertisers to gain the most from this kind of analysis, as I will discuss below, Web publishers will need to have the kind of deep knowledge of their visitors that will produce useful advertising insights.
Who will produce and verify the data?
"Probably not comScore or Nielsen," says Peterson. The problem is the panel approach. Projecting results from a small sample of users is increasingly controversial looking at simple metrics such as reach and composition. It will not work for engagement, as we have described it here. To get at the depth and complexity of this engagement model, we need to look at publisher data. As we discussed in the first article in this series, the startup firm Quantcast holds special appeal. Quantcast has advocated a methodology that normalizes direct publisher data through its "mass inference" algorithm. When I outlined this approach to CEO Konrad Feldman, who views the work of Eric Peterson "with the highest respect," Feldman said he believed that his company could in fact produce this kind of indexing. Are there other companies who can take the engagement model forward? I welcome all nominations.
How can publishers collaborate with advertisers to yield actionable intelligence?
"We've developed Web technology to the point where we have an astounding wealth of data about audiences. Publishers can tell us what content audiences are consuming and the share of content downloads among competing advertisers. All this has been great. But what does it all mean? How can we turn that information into something we can act on?"
Brandon Starkoff, Vice President/Global Director at Starcom Worldwide
Starkoff's point is critical to the whole point of seeking to establish a definition and a set of metrics for engagement. What does it matter if, as far as media companies are concerned, it doesn't produce better insights into what will make advertisers successful with even the most "engaged" audiences? The kind of audience knowledge Starkoff says he is seeking is "predictive intelligence-advice on what kinds of advertising will work with a particular audience or audience segment."
Advertisers right now think about engagement as a way to distinguish sites from each other. David Smith, CEO of Mediasmith, a San Francisco-based ad agency, is also a board member of Quantcast. When we looked at two of the largest sites in terms of traffic volume, Facebook and About, Smith pointed out how over 60% of Facebook's users are returning to the site more than 30 times per month, while just 2% of About's visitors come back with that kind of frequency. "That's why the entire advertising community is trying to figure out how to connect in with that level of engagemen," Smith says.
The Peterson model, I would argue, points us toward that kind of intelligence. We would be able to understand exactly what engagement means in terms of interactivity, content consumption, content generation, and loyalty. Publishers can show what kinds of content and what forms of activity make up those indices. Using analytics programs, such as WebTrends Visitor Intelligence and Score, for example, an automotive category publisher can provide detailed insights into the most engaged audience segments and show the likelihood visitors will respond to a newsletter on car audio accessories, a how-to video on hooking up MP3 players, or a Web site community blog.
As an industry, we are a country mile away from predictive intelligence. No doubt the tools are available and there are no real obstacles for individual publishers to take the entrepreneurial path of showing advertisers how to best target their audiences. I wouldn't be too surprised to hear that there are a good many taking steps in that direction. Yet there is a media industrywide opportunity to better understand engagement, establish a definition, and to agree upon standards. It is my hope that this series contributes toward that end. Please feel free to offer your ideas.
1 person recommends this article.
RSS FEEDS
What your audience does outside of your site on the web can actually influence your engagement levels and type more than the intrinsic nature of your site. This also seems to influence marketing campaign engagement on the site – what the user does outside of your site may influence ad effectiveness on your site more than the intrinsic nature of your site. There are some obvious parallels with Behavioral Targeting here but it goes much deeper than that, at least in terms of how most people understand BT today (or more likely, the lack of understanding).
What does this mean for publishers? If you truly want to understand your user engagement model, you need to understand what your users are doing outside of your site, relative to what they are doing on your site. You can then use that understanding to a) better leverage your existing engagement model, b) alter your engagement model, c) better monetize your site. There are some serious challenges here for most web publishers but they are not insurmountable and well worth the effort.
I hope that there are others looking at user engagement from a ‘network’ or web wide perspective – I would love to discuss this topic more in depth with others that may have already tread down this path.
Pure analytics guys would not be happy with you trying to measure engagement as they would say it’s a tactic, not a metric. However, as a both an analytics and marketing guy, I like how you’ve negotiated between the two.
I sent you Avinash’s Trinity of analytics model and I indicated how I was very influenced by his clarification that the click stream data is only the “What� and you have to extrapolate the “Why� from User Centric Data such as opinion popups, A/B testing, forms, etc. However, I think your articles have shown that we can actually figure out some of the “Why� using the “What�-data…who can argue that Facebook users aren’t more engaged than About users based on the return visits metric? Lol, we don’t need to ask them, we can see from the data, right? Yes, it’s an assumption, but I’m on board.
So, I guess this continues to be filed under the “what we think we know about online behavior�. As I tell my colleagues, online marketing insights are kinda like building a deck of cards where every data point that can be backed up is like a “thick card� and every wild assumption is a “thin card�. We’d like to build all of our house of cards with thick cards, but sometimes we can’t. That’s where the right-brainedness (is that a word?), creative reasoning, and experience take over and we can live with the fact that our assumptions, although not 100% supported by data, could be in fact thick cards as well.
I’m big into nomenclature and now that you’ve put some of these metrics ideas into words, I’ll be able to start testing your hypothesis in my daily work; specifically, the way “recency� and “interactivity� definitely have a place in my world. I’m a member of the Web Analytics Association Standards Committe and there are a lot of statisticians that just won’t get what you’re trying to say. However, I know our agency has been looking for ways to better measure the ways users consume our clients’ media so, once again, thank you for pushing the discussion outward.
-JD
And in that respect comScore and Neilsen are fair game because they do indeed attempt to measure a site's performance in terms of engagement. In March of last year, for example, comScore announced its new suite of "visits" metrics as a "comprehensive" approach to engagement.
I am thinking too about your comment that we need to be careful of a linear approach to engagement--and couldn't agree more. What I love about Eric's formula is that you can see engagement from multiple perspectives. What the right weight for each? I have ideas that I am continuing to develop through my own work and in talking with others--or perhaps from other comments on the MediaPost site!
Now, if an individual site decides that clicks are more important to them, in the context of my framework they have great latitude to change the weights. The only thing they need to be careful of is to ** explain ** to readers how and why the weightings are applied. This is less of an issue in my mind given that ** ALL ** web analytics and measurement data really calls for some type of explanation, especially when used in mixed audiences.
Regarding your second suggestion to "get our house in order" first, you're preachin' to the choir here (try Googling "eric peterson" cookies) This is, ironically, why I retracted my comment about "probably not comScore or Neilsen" ...
While panels may not be the ** best ** way to project the actual number of visitors to a site, I think we can all agree that cookie-based methods are no more perfect in light of the data I first presented in 2005 while at JupiterResearch, and the data that comScore more recently added to the conversation. But, comScore's ability to gather accurate information about long-term visitor behavior theoretically far exceeds that of the browser cookie.
This is obviously valuable because the visitor engagement calculation is far more robust when calculated over the lifetime of user interactions with the site. And while it's kind of difficult to explain in a blog comment, being able to examine the ebb and flow of visitor engagement by source, page, campaign, etc. is hugely valuable, especially when you consider offline marketing efforts.
That said, I'm not sure the cookie deletion problem can be solved. In fact, I'm positive that it cannot be solved. So I guess I disagree that we should exclusively rely on useful yet less robust like "bounce rate", "conversion rate", and "average time spent" when a new model is (rapidly) emerging.
First, I think it is improper to say that "comScore and Nielsen methods fall short due to a fairly superficial approach to engagement". Both those reputable companies are primarily audience measurement system companies (AMS). They do not purport to be measuring 'engagement' so it is unfair to critise them because they don't.
Second, while your model is interesting may I make a few observations. I agee with the point made by Chris about medians given the 'long-tail' skews. However, there is an underlying assumption of linearity in your model. Virtually every model I have ever built (media, retail etc) end up being non-linear ... which is basically the way the world works. There is also the assumption that the six factors you talk about are of equal weight, which is a naive assumption - though it may well be right. The factthat you have settled on six factors is good as we want as parsimonious a model as possible.
But the deeper underlying question is what 'engagement' are we trying to meaure. At the uppermost level each medium has a different level of engagement - for example, in most of the studies I have seen magazines are at the top of the list ... cinema is up there too. Billboards and ambient media tend to be down the list.
Then within each medium each "vehicle" has it's own engagement. Vanity Fair may have a higher engagement than Hello. CBS may have a higher engagement than Fox. Facebook may have a higher engagement than MySpace. The list is virtually endless.
Then within each "vehicle" each "property has it's own engagement. Brothers and Sisters may have a higher engagement than CSI. The list is even longer.
Then, at the bottom of the hierarchy, is the client's ad. Yes, the very money that keeps this whole system going! The actual ad itself has an engagement factor. There are 'good ads' and 'bad ads'. Placing a good ad in a bad medium/vehicle/ property will make the ad grossly underdeliver.. Placing a bad ad in a good medium/vehicle/property will not 'save it' ... a bad ad is a bad ad. Obviously we want good ads in a good medium in a good vehicle in a good property.
My instinct tells me that there is an effectiveness hierarchy in there as well. The medium's engagement is swamped by the vehicle which is swamped by the property which is then in turn swamped by the ad. I have no proof of this - but it does make sense.
At the end of the day the person paying the bill really wants all four measures ... but i still think the bang for the buck lies in measuring the engagement of the ad. Let's get that right and work back up the hierarchy!
One other thing. You mentioned that "over 60% of Facebook’s users are returning to the site more than 30 times per month". Is this actually 60% of users ... or 60% of cookies. These are VERY different things. I suspect that it is cookie based on a web analytic tool - none of which I have seen take into account cookie deletion over a period of time (correct me if I am wrong).
Cookie deletion over a month overstates the REAL audience by around a factor of 2 to 2.5. Take for example the data for Australia. The online 'audience' is over 35million. Problem is we only have a population of 21million of which 75-80% are online in a given month. To be fair, this factor comes down to around 1.5 to 2.0 for major sites and portals - yep audience is a little more than half what is reported. And when you drill further down the effect can be as little as 10% on small niche sites.
Josh - you may like to comment on youir findings as well.
My humble suggestion is that the industry needs to get its total audience numbers sorted out first, which in my humble opinion WILL require a panel of some sort integrated with server based data. After all, if it was commonly reported to advertisers that 200% of Australia's internet-capable population were on-line in any month what do you think their reaction would be. Once we've conquered that, let's have another look at engagement.
Cheers,
John Grono GAP Research Sydney Australia
And Eric, wouldn't you think that comScore or Neilsen could take an engagement index (with the necessary complexity we have been discussing in these 3 articles) only if they utilized publisher data to complement the panel-only approach now in place? This is why I have suggested that newcomer Quantcast might be in a strong positon.
Imagine a series of vertically focused engagement calculations --- one for media, one for retail, etc. --- in which thresholds were set based on average/median/observed distribution of participant data across the core index calculation. Media buyers would then be able to buy at "high engagement media properties where engagement is primarily driven by loyalty and likelihood to subscribe," etc.
Furthermore, if someone like comScore (per Mr. Chasin's comment above) were to develop this engagement metric, the census-derived data would still give site owners increasing granularity into engagement since they would have a better, more personal view into the Interaction Index, the Feedback Index, and more relevant thresholds for all indices used in the final calculation.
Anyway, thanks so much for bringing the calculation to light here at MediaPost Kevin.
One minor suggestion for your approach and Peterson's model, though. I think industry medians should be used as reference points for individual indexes that make up the composite engagement index (instead of averages or means). In my experience, I've found that medians provide a much better gauge of central tendency than means, especially when data distribution follows a non-normal or non-bell-shaped curve (as I'm sure you're aware almost all Web data distributions follow some sort of skewed distribution curve).
Keep up the good work!