In Algorithms We Trust - Too Much?
Do we trust algorithms and mix-modeling studies too much? I recently met with an advertiser who said the reason that radio wasn’t part of their marketing efforts was because it didn’t perform well in their mix-modeling analysis. And they had no intention to use it moving forward.
When I asked when the analysis was conducted, the executive said “four years ago,” which predated this person’s employment. What this executive didn’t realize was that radio had actually played a huge role in the company’s initial success. Clearly something was amiss. After further review, it became increasingly apparent that the mix-modeling study might not be up to 2013 modeling standards.
First, this advertiser's modeling study only quantified what is referred to as "direct" impact, which measures the sales that occurred as a result of using a specific marketing channel. This is not ideal, as failure to quantify a media channel’s “indirect” impact paints an incomplete picture. For example, "indirect" impact is when a consumer is exposed to a radio commercial, later Googles the advertiser, clicks on their link and makes a purchase.
Mix models that are incapable of teasing out indirect impact increase the chances of an advertiser implementing a marketing plan that is skewed toward paid search, couponing and promotion. An algorithm that focuses exclusively on direct impact will not measure radio's key role in the purchase process above.
Second, it’s also important that a modeler's algorithm be sophisticated enough to quantify one medium’s impact on another. This synergy between media, as consumers engage with multiple forms of entertainment, often occurs simultaneously and must be accounted for in the form of a “multiplying effect.” It happens when exposure to a commercial in one medium triggers recollections of messaging that were initially generated by another. This must be measured to get an accurate read on each medium’s total delivery. It too was excluded from this advertiser’s mix-modeling algorithm.
Lastly, this mix-model study focused exclusively on short-term channel performance. Near-term focus will always skew a marketing plan toward paid search, couponing and promotion, as it is not at all unusual in the short term (2-6 months) for media channels to produce a ROI of less than a buck. To be able to grasp any channel's total impact, both short- and long-term (usually 6 months-2 years), must be considered.
This advertiser and others we have recently met have lost sight of the fact that mix-modeling results are not intended to be blindly followed or forever binding. Instead, the objective should always be to seek to understand and reveal why a particular medium performed the way it did. Creative, scheduling, GRPs or any exogenous variables -- such as the economy, competitive marketing or offerings or even the weather -- can all impact mix modeling’s results. Advertisers should not view mix-model results as black or white -- and do so at their own risk.
It is also a good idea for those interpreting the mix-modeling results to ask themselves whether the results pass their “logic test.” The results of this particular study should not have passed due to its unprecedentedly low radio metric and previous performance history.
Mix modeling should not be the only measure of performance. The role each medium plays in a campaign tends to differ depending on which part of the purchase funnel it excels in. Long-term brand-building is critical. Performance can be measured by other means, such as tracking studies focused on brand health metrics including awareness, intent, Web site visitation, test labs and even control vs. test studies. These results should always be compared and balanced against mix-modeling data results.
It is understandable that there is pressure to act upon mix-modeling results, if for no other reason the significant cost of conducting these studies. But a balanced approach is critical to delivering the best results for the short and long term. Logic, common sense and the desire to understand the modeling results will always lead to a more effective allocation of marketing resources.