Commentary

'Any Idiot Can Optimize One Variable'

by Kyle Johnson , September 1, 2009

That was how one former colleague expressed his frustration when a new employer framed all the company's goals in terms of new customer orders. A predictable consequence followed: managers dug out every kind of discount packaging ever hatched by the JV sales team in pursuit of scoring orders; and lo and behold, new customers came and ordered -- almost as quickly as profit margins fell. It's easy for most of us in online advertising to chuckle at a story of one-metric management, given that as a group we have more performance data than any other group outside of the Federal Reserve.

In fact, thanks to the measurement systems of publishers, agencies, advertisers and Web measurement companies, we can analyze the success of our ads and campaigns seven ways to Sunday and still have new options left for Monday. Reach. Reach in target. Increase in brand awareness. Clickthrough. Viewthrough. Conversion. eCPM. Cost per action. Estimates for lifetime customer value. Ongoing measurement of customer value. The list goes on, and all these metrics can be delivered fresh to your inbox, daily.

Even with the plethora of data available, how many planners and buyers focus only on one metric when setting goals, judging success, and optimizing? Surprisingly, the majority of them. The metric of choice varies, especially across industries -- retailers set their compass by cost-per-transaction, while a consumer electronics manufacturer might dwell exclusively on unaided awareness -- but devotion to a single metric is very common and very dangerous nonetheless. Sure, narrowing your measurement focus to one variable reduces opportunities for confusion, delayed action, and for decision-making hinged on too much complexity. However, taking this mono-metric approach also shuns the potential power of the data at our fingertips, leaving marketers blinded to new opportunities and ill-equipped to achieve our objectives.

Two metrics are better than one

If online advertisers entirely lose ourselves to one metric, then we aren't as smart as we could be. This is most obviously true when devotion to one metric leaves us ill-equipped to act on a secondary objective. Consider the media buying group of one advertiser where the metric of choice is cost of acquisition, and media planners are also directed to make "strategic" buys on key sites. Once that word "strategic" is invoked, planners not only get to set aside devotion to the single metric - they get to set aside reference to any metric whatsoever. The result is.... well, nobody can tell what the result is.

There's growing awareness in the industry that all campaigns inevitably yield both direct response and branding value. Given all the data that's available, it's critical that media teams evaluate success in at least two dimensions. A baseball team has to rate its shortstop both as a batter and as a fielder. Likewise, the direct-response advertiser should still have a measurement plan and process that evaluates branding value, even if only by a gross measure like exposure.

One metric also becomes a barrier to success when it leaves us out of touch with the other variables that can be directly influenced or accurately predicted. We don't always get to choose cost-per-action, after all -- instead, we make changes that can favorably impact rates at different points along the path to action. We need to maintain a view one level deeper -- e.g., of both open rates and take rates on e-mail campaigns, since those are variables we can truly impact by changing subject lines and calls to action. We all know this, perhaps, but do we always do it? Do we do it routinely enough to know whether open rates or take rates are easier to influence given the audience and offer? It becomes all too easy to simply repeat what's produced the highest "score," and to lose our grip on the levers that might enable even greater success.

Leaving opportunities on the table

Afflicted by a case of mono-number-osis, online advertisers also lose the ability to make new discoveries -- both about underlying bellwethers that might be open to influence and prediction, and about the shortcomings of our current charting. We all know that simply taking a longer-term view of the "one metric" will often necessitate a revised evaluation of campaign choices. A longer-term view of several metrics might reveal that increases near the top end of the funnel (reach, brand awareness) do materially contribute to success on the bottom line.

By going even deeper and by looking at more discrete metrics, it's possible to make truly tactical, actionable discoveries - perhaps the key indicator of long-term "direct-response" impact for a "branding" ad is the amount of research on review sites and queries on search engines stimulated by the ad. In the research and measurement business, we uncover surprises like this all the time.

Baseball players don't improve by deciding to hit more, they improve by discovering a better way to swing. That is to say that we can't help but manage to one metric sometimes; and we shouldn't stop trying to look for a single metric that could serve us better.

Maybe one company moves from short-term, cost-per-action to a metric that takes lifetime value into fuller account. That kind of innovation, the development of a new, better shorthand for success wouldn't be idiotic. It would be efficient, effective, and smart.

But standing pat on that one new metric, forgetting the elements that feed into the new metric and facilitate action and prediction limits the success of campaign -- that old colleague might say it's not a very wise idea. We have so much data, and so many numbers, and they contain so much value. Let's not forget that we have to keep more of them on the table and triangulate -- not just calculate -- in order to outfit ourselves for greater success.

metrics

6 comments about "'Any Idiot Can Optimize One Variable'".

Check to receive email when comments are posted.

Jeff Greenfield from C3 Metrics, September 1, 2009 at 2:27 p.m.
So true Kyle that today's online marketers have a plethora of metrics .. of course none of those mean anything if the person doesn't even see your ad.
That's right -- below the fold ads measure just as well as top banner position.
That is ... unless (and here comes the shameless plug) you are using C3 Metrics.
Reply

Nance Rosen from NanceSpeaks!, September 1, 2009 at 2:29 p.m.

If you use Harvard's Balanced Scorecard approach, you have at least four dimensions/criteria for evaluating outcomes: they could be the impact on financial objectives, human asset performance, training and development and R&D. Of course, you can add your own outcome criteria. Social media is going to demand we, as marketers, look at outcomes that include sales but go beyond that - otherwise, we will destroy the communication purpose and opportunity of social media, and make it simply another form of FSI (free standing inserts) hawking discounts for Dove soap (no offense Dove soap, I use you).

Jonathan Hutter from Northern Light Health, September 1, 2009 at 2:30 p.m.

Excellent column, it really opened my eyes. The headline is something I often say about golf (I can hit a drive a long way), "any idiot can hit a driver."

However, even with all the data inputs available in golf -- driving distance, fairways hit, greens in reg, putts, etc., there is still one final measure, score.

Likewise with campaign metrics, all of the data inputs still have to lead to a final business metric. Without knowing that, you just continue to play the ball on a course with no holes.

Brian Ferrario from Rocket Fuel Inc., September 1, 2009 at 3:05 p.m.

bravo -- bring on the data. i was trying to think of a good line -- something about in the land of the blind the one-eyed (one metric) man is king. in racing they talk about "target fixation" -- fixating on one thing. losing the racing line, and almost always crashing. could be the same way with metrics. advertisers need to learn not to fixate on one metric, but to build a hierarchy or quiver of things to be open to monitoring, measuring, tweaking, etc. for both DR and brand campaigns (meaning real-time sales and longer term brand lift goals). and they need to learn that there's probably an optimal mix to experiment with -- meaning they are all interdependent variables. we've run awesome campaigns that have moved the needle on an advertiser's business, but somehow they were fixated on some metric that might not have lived up to their expectations, even though the overall campaign did a number of other amazing things. and sometimes, they've kept the real metric they're measuring a secret -- never telling their partners what they were measuring and who delivered the best value -- how does that help build strong relationships that will eventually make a brand stronger?

John Grono from GAP Research, September 1, 2009 at 6:51 p.m.

To continue the baseball analogy, I would suggest that there is really only one metric that the teams, team owners and fans care about - winning the World Series. You can have every individual leding their stats on errors, hits, RBIs etc and not walk away with the prize.

To give a recent example - painful to all Aussie cricket lovers - Australian recently played the 'old enemy' England for The Ashes. There were 10 century makers (scored 100 runs in an innings) across the series, 8 of which were Australian. Australia had the three top wicket takers. Australian scored more runs. Australia took more wickets. Australia had better bowling and batting averages.

Guess what. England won The Ashes 2-1. Yep ... the only metric that all the Poms and the Aussies cared about was who held up the Ashes urn at the end of the series ... and sadly it wasn't the Aussies.

Jeff Zwelling, September 1, 2009 at 7:01 p.m.

While I agree with Kyle's premise, I'm also a big fan of determining a singular metric that everyone on the team can rally behind. That doesn't mean that I don't collect every metric under the sun, it only means that I make sure that all the marketing activities we do are being compared apples-to-apples, regardless of whether the goal is branding or DR. In the alternative, there's a human tendency to focus on whatever metric makes the specific marketing event looks best. For example, I've seen too many people justify their non-ROI positive email campaigns by focusing on open rate and click-through. But ultimately, the allocation of resources to that email campaign must be judged by the eventual revenue it generates - meaning ROI.

So what's the singular metric I focus on? Revenue per visitor (at a campaign level). It gives credit where credit is due. So rather than only crediting that email campaign for the direct revenue that was generated from immediate click-throughs to the ecommerce site, Revenue-per-visitor allocates revenue back to the marketing action that eventually results in a sale. If the Sept. email results in a bunch of visits that ultimately lead to a bunch of sales, the email campaign gets some portion of that revenue. Marketing channels that generate high RPV get re-invested in - those that don't - get dropped.

The challenge of course with this approach is being able to track all of the marketing events that led up to the sale, regardless of cookie duration, cookie deletion etc. This means that you have to have some outstanding tracking in place, which few marketers do. Hence, the band-aid approach of finding metrics that you can collect as proxies for the metrics you should collect.