• by David Goetzl  @dgoetzl, July 5, 2013

Some might say with so much media research promising improved insight into ROI, the role of marketing mix modeling (MMM) has been overlooked. Recently, though, the trade's complicated work has received some criticism and come under increased attention, largely through a report by an industry group.

The report on the state of MMM came from the Council for Research Excellence (CRE), which commissioned consultancy Sequent Partners to conduct a study. CRE, which says it acts independently, is funded by Nielsen.

Both Nielsen and Sequent have a stake in how MMM evolves at agencies and clients. Sequent showed a commitment to independence in its research, though, by offering disclaimers about potential conflicts of interest on multiple occasions.

The CRE report, completed this spring, determined that marketing mix practitioners have a “great deal of integrity and pride” and are “sensitive to criticism about their ‘black box’ legacy and a perceived lack of transparency.” The research determined much of their “statistical techniques” go over the heads of clients.

But it credited some modelers for pushing the envelope on evaluating how to navigate building short-term sales growth along with long-term brand equity. Their work has become more complex with the onset of social media and boom in mobile use. There has also been growth in big data and single-source research, which looks to link media exposure with purchasing behavior.

The report concluded that agencies could play a greater role in the modeling ecosystem. Their involvement is “considered better than it was 10 years ago, but still not where it needs to be.” One potential issue is clients use so many different agencies for so many different disciplines that coordination can be difficult.

MMM works best with "a cross-functional team and consistent involvement from the marketer, the modeler and the various agencies,” the report said.

The research included interviews with some 50 modelers, advertisers and academics. Modelers came from firms such as Ipsos MMA and Accenture, while advertisers included researchers at AT&T and Staples.

Among the modelers was Art Thomas, a veteran of MMM who worked at IRI and The Modeling Group before moving within an agency structure. He now leads the advanced analytics group at Mediabrands' Universal McCann, where he’s worked with a client list that includes Hyundai, MasterCard and Six Flags.

In the spring, Ad Age ran a piece detailing some of the criticism percolating around MMM. It cited CBS Chief Research Officer David Poltrack suggesting modeling practitioners have failed to sufficiently incorporate newer single-source research into their work and “attribute sales to specific segments or groups of consumers.” Poltrack also suggested modelers overlook the impact search can have on sales with its potential for immediate action.

It should be noted Poltrack would hope modelers find TV and radio – core CBS businesses – make for exceptional uses of advertiser dollars.

Ad Age also noted Effective Marketing Management’s David Hoo believes MMM has led consumer packaged goods (CPG) companies to misguidedly move money away from advertising into promotional activity.

With all the interest in MMM, Universal McCann's Thomas spoke with MediaPost and offered his take on the criticism and other issues affecting the business. His edited comments follow:

On suggestions modelers haven’t moved fast enough to integrate emerging marketing options into their optimization models:

"Just like media has been evolving a lot -- especially over the last 10 years -- we expect there to be more people really looking into what the models are showing. So a lot more interested parties. Our clients are very aware of new media channels and there’s always excitement around what’s new. You’ve got to make sure that balances with what’s been around for a long time and what’s still very reliable, even if it’s considered I guess traditional media.  

I’ve seen so much change just over the last 15 years and I expect there to be a lot more change going forward because (MMM) is an industry that really has a lot of people very anxious to explore new avenues (and) to look at better ways to read things. 

To hear criticism like that … some of us agree with it to a certain extent. I certainly do.

I know there’s always a risk of misattributing something to a short-term outcome and that’s something we’ll have to keep in mind. I think a good modeler, a good practitioner knows that, but there’s always a risk of somebody misusing model output for sure.”

On the Ad Age article saying some critics believe models fail to acknowledge the value of advertising, including network TV, and overly focus on promotions around price:

“I’m not surprised. I think there is a danger of that happening. As a modeler, you have a couple of fears. Number one, of course, is you do a model and it sits on the shelf and nobody pays attention to it.

But the second fear and I think it’s actually much more dangerous is the idea that (clients and planners) will take the model results as gospel. They’ll go back and when you’re not there to help them interpret the results … just run with it.

There is a danger of seeing big returns from immediate actions like promotions and couponing for something that’s a CPG (product).

A standard disclaimer we use for every presentation we give -- and every time we’re talking about the ROIs with clients and planning teams -- is if you’re trying to hit short-range goals, you can do that through promotions and through couponing. But, there are a lot of risks you run there. You never want to train your consumer to wait for a promotion.

The other thing is you can kind of cheapen your brand image a little bit by looking like it’s not necessarily worth your real price you want to sell it at. And so we make sure we make that clear to planners and down the line to clients, because really your brand equity is the most important thing.

If used correctly and judiciously, promotion is not a bad way to get people in there, but you never want to rely on that as your main marketing effort. I understand the concern there for sure.”

On suggestions modeling fails to account for the benefits of search enough:

“I think it’s false. We have trouble capturing search really well in the models because it’s always out there and that can be kind of a weakness for a time series model. But I don’t think we’re ignoring it. I do think we understand that it’s there and I think the fact is we use search very often in those upper funnel metrics, in those awareness metrics and consideration models.”

On MMM’s evolution and changes coming, including with social media:

“We’re not necessarily modeling against sales these days. We do run sales models, but we construct an overall modeling structure that looks at the entire funnel.

There are so many more metrics that are available now than 10 years ago. We’re measuring awareness, we’re measuring top of mind, we’re measuring preferences and we’re measuring recommendations -- kind of post-sales satisfaction measures. It’s about getting people considering the brand and then getting that conversion anywhere along that cycle.

And, then on the other side of the coin, when it comes to actual metrics, there’s a lot more interest now in social media and just online media in general. There is the question of how can you best capture the impact of advertising and other marketing on social media -- and whether that’s (measuring) brand equity effectively (or) whether it’s just kind of a thing that people are not really involved with as much as we think they are. So, the challenge there has been making sure we’re measuring the right metric for social and I think that’s (gotten) better and better over time.

I don’t trust a lot of the metrics right now. Because Facebook likes? What does that really mean? Is that really involvement by a consumer? I don’t really think it is.

We have Twitter mentions (about a) product. That seems to be a little more accurate, but we’re still kind of working on it. There is a lot of noise in that sphere. We want to make sure we’re looking at the right data to make sure we’re measuring things correctly.”

On what new metrics provide as far as broader insight:

“A much better picture of the entire realm for a product … If you have a big purchase like an automobile or something like that, it’s a very long (period) between when (a consumer) sees an ad and considers a product and when you actually go and make a purchase. And now with better metrics (capturing) those in-between steps, we’re able to get a much better picture about how a TV ad for instance … feeds the awareness part and consideration part before you get to actually deciding on that final purchase.”

The best people in this profession tell an entire story. They don’t just give an ROI and an effectiveness and efficiency measure. You give a whole story about how a consumer considers a product. So, it gives us much more depth that way.

We’re talking about getting people involved and moving them down the funnel. When it comes to planners using the modeling results … when you’re trying to drive awareness, if you’re trying to get trial for your product, the most effective way may not be the same one that necessarily reflects itself very easily in the sales down the line.”

On cross-platform measurement and its intersection with MMM:

“Part of the challenge going forward is how do you really measure that ... One thing we have to remember, we really need to keep in mind throughout the modeling process, is that cross-channel can make certain things look more attractive than they really are compared to other things.

So, something that happens a lot is you’ll see digital video doing very well and digital in general doing really well -- because very often it’s much closer to that final result, that KPI you want to measure. But it shouldn’t come at the expense of more broad-reaching media – your network TV, your print ads. A good campaign tends to strengthen all of those things.

We’ll (give) a presentation that says your effectiveness and your ROI is higher for some of these new digital advertising platforms, but those wouldn’t work as well without a really solid broad-reaching TV ad. That is the foundation that allows these more targeted media channels to be as effective as they are. So, it’s really important to get a message out that people are familiar with, that kind of resonates with the consumer. And, then you’re able to go in and hit them on some of the smaller media platforms and make those so much more effective.”

On the impact of single-source measurement products like those from Nielsen Catalina Solutions and TRA on MMM:

“I think overall it’s helping. Anytime a client understands the usefulness of analytics in any way, shape or form as a tool in the entire portfolio of their planning process, I think it’s good.

I don’t know if marketing mix is ever really going to be replaced. I hope it isn’t actually. But I do think it is going to be incorporating more of these other strategies over time.

I’m always amazed at how many fairly large clients haven’t used any analytics over time or very few anyway. And, then you show them the process and their kind of like wow we’ve never really had numbers that back up some of (our) decisions …

I don’t think it’s a bad thing to at least make them open to the idea of using analysis as a way to at least guide some decisions -- not in a black and white gospel kind of way -- but at least a way to kind of nudge them one way or another.

Part of the whole thing is kind of educating clients and making them very comfortable with the numbers, with the process, with the math behind it, without necessarily going into depth about the math -- but at least having them comfortable with the idea that you know what you're doing and that you’re using both math and reason in combination to hopefully inform some good decisions. The more players in the market, I think the better it is.”

On the most impressive new metrics available:

“Better tracking of social media has been great. It’s really helped fill out that brand profile really well. And, more granular data overall.

The more granular data overall has made things much better. The trick to our industry is variation, making sure we can measure changes in inputs. And anytime we get more granular data -- whether it's digital, whether it’s radio, in some cases TV is still getting better about that -- the better it is for us. I think across the board most media channels have gotten better about that over the last few years.”