• by Douglas Quenqua , December 29, 2010

The reliability of focus groups has been questioned nearly as long as they have existed, but social media data mining is fast becoming a trusted way to learn what consumers think

Why start a conversation about your brand when you can eavesdrop on one - or, for that matter, one million - already in progress? That's the premise behind social network data mining, the process of listening in on consumers' brand-centric conversations online, then synthesizing those conversations into actionable market research data.

Forget paying not-quite-randomly selected customers to sit in a focus group for an hour and pretend they don't know you're watching from behind the mirror. And forget making thousands of calls in search of a few dozen people who'll sit through a 10-minute phone survey. Social network data mining - or, more simply, social media research - provides a vast, candid peek into what your customers (and potential customers) say about your brand when they're not being paid or monitored.

"Social network data mining is growing really fast, because it gets the closest to what clients really want," said Ray Poynter, author of The Handbook of Online and Social Media Research, "which is to hear their audience's voice when they don't know they're being listened to."

To its boosters, social network data mining is the fulfillment of a promise made at the dawn of the Internet age: that traditional focus groups would be replaced by public conversations spontaneously taking place online. To its critics, it's the latest in a long line of tactical advances that still fails to correct what's wrong with market research. (More on that later.)

For now, it's hard to find someone who is doing more to advance social media research than Annie Pettit. Pettit is chief research officer at Conversition Strategies, a boutique research firm based in Toronto, and one of a handful of people working to refine the practice and convince chief marketing officers of its value. Like most true believers, Pettit talks about social research in fatalistic terms. "I'll stake my life on the fact that this is where research is going," she says.

But dismiss her as a zealot at your own risk. Pettit, who holds a Ph.D. in experimental psychology and boasts more than 15 years as a practicing researcher (she wrote her dissertation on methods for ensuring the integrity of online research panels), has spent the past two years refining the techniques that have made Conversition the company that Fortune 500 companies go to when they want to mine the social Web for marketing data.

"A lot of companies that do mining collect the data and put it on a dashboard and let you watch as the volume of data goes up and down so you can see who's saying what," she says. "We go the next step and do all the sentiment analysis, so we determine if someone is saying something positive or negative or something in between, and we get a range of scores so we can tell you how positive or negative."

Pettit is also helping develop techniques for divining the answers to classic focus group questions from the content found on social media. This way, companies can not only find out what customers think of their brand, but how those customers might react to changes, or what changes they would like to see. "Traditionally, market researchers care about questions related to buying and intent to purchase," she says. "We have replicated those same questions on our platform ... Today, you could take an online survey and basically map it question by question onto social media research," she says.

The idea of using social media to monitor one's brand has been around pretty much since the dawn of social media. These days, any company not monitoring Twitter or blogs for mentions of its brand has already fallen behind. This has led some companies to conclude - falsely - that they are already doing social media monitoring on their own.

"Just because you buy a scalpel doesn't make you a surgeon," says Ed Erickson, president of consulting firm Erickson Research in Chicago. "Mining through these things even with some of the tools that exist is a pretty labor-intensive process. Also, somebody has to be able to read the patterns they are seeing."

As such, the need for researchers who can dig deep into social media and interpret those findings is likely to skyrocket in the next few years, and revolutionize how marketers test their ideas, according to Pettit. "The idea of using social media to replicate research is absolutely brand new," she says. "It's only this year that people have kind of considered it, and I think by the end of two years time we might have people on board with it as a legitimate form of research."

It'll be faster than the last big change in the field. "After all," she notes, "it took people five, 10 years to get on board with online surveys."

The search for something to replace - or at least augment - traditional focus groups is nearly as old as focus groups themselves. Ask any marketer who's spent enough time hiding behind a one-way mirror what he or she thinks of the focus group process, and prepare yourself for an earful of gripes: People aren't honest when they know they're being watched; geographic limitations make it hard to gather truly representative samples; one or two dominant personalities tend to influence the group's opinion; paid incentives attract "professional" participants; unskilled moderators can exhibit a bias that poisons your findings.

Ultimately, all such complaints boil down to one nagging truth: Focus groups do not measure what people think when they are making a purchase; they measure what people think when they are participating in a focus group.

"A focus group really is not a realistic situation," says Erickson. "It's pretty stilted in that you're taking these people out of their normal lives, sitting them down in a room and telling them they're being filmed and asking a bunch of questions." Erickson's comments call to mind the Heisenberg Uncertainty Principle (or at least the popular understanding of it): An object can never truly be known through observation because the act of observing changes the object.

Today, while the traditional focus group is far from obsolete, the Web has vastly expanded the means by which marketers can solicit opinions from consumers. "It's extended the way we can talk to people and the way we can observe them," says Sheron Davis, senior vice president and behavioral sciences director with BBDO. "It allows us really to fulfill a number of objectives in a much more sensitive way."

Social network data mining is just the latest of these new methods. The first and most obvious was to begin conducting focus groups online, which relieved the process of some serious disadvantages. "Online focus groups give people the opportunity to organize by market segment, rather than geography," says Linda Stegeman, president and founder of Fremont, Calif.-based Artafact, which makes software for online focus groups. Now, if a pharmaceutical company is looking to gather opinions from people who suffer from a rare disease, it can do so at considerably lower cost and with much greater frequency.

Stegeman also claims there is less peer influence online. "When the moderator asks a question, everyone answers at the same time" because they are typing and not speaking, she says, preventing any one person from dominating the conversation and influencing others' answers.

"We refer to is as cost-per-verbatim," says Peter Mackey, executive vice president of research at Invoke Solutions, a Waltham, Mass.-based company that facilitates online focus groups. "You not only have more people participating, but you have more people answering at a high level, whereas in traditional focus groups, you might get two or three people who are dominant and everyone else sits in a corner and you have to pull answers out of them."

Client participation is greater as well with online focus groups because no one has to stuff themselves into a tiny room behind a mirror; an unlimited number of people can log on and watch the session.

The rise of social media is opening new doors for researchers, too. Over the past few years, many marketers have been replacing focus groups - online or off - with their own private social networks known as brand communities. Here, they can gather customers together to give feedback on products at their leisure and in their own language. Marketers from Rubbermaid to Pepperidge Farm are using brand communities to test new ideas, but perhaps the most popular example is Starbucks, whose My Starbucks Idea Web site has yielded dozens of product ideas and fostered true fandom among customers.

Because these communities don't require consumers to show up at a particular place or even time, brand communities have vastly expanded the number of customers from which marketers can harvest opinions.

Still, brand communities "share a lot of the same downsides as you have in traditional qualitative research," says Erickson. "You're essentially puling the people in for the express purpose of soliciting their opinions."

Because both online focus groups and brand communities still require people to devote time and attention, most of them require marketers to "incentivize" participants either with cash or products. Those incentives occasionally produce "professional" survey takers, people who are skilled at getting themselves into focus groups, or who take every survey they can find, simply in order to reap the rewards. Many of these people fake their identities to get on panels in the first place.

Conveniently enough, the Web is offering solutions to that glitch as well. Last year, the CMO Council came up with a model that removes the temptation of personal gain while preserving the incentive to participate.

Called Pause to Support a Cause, the program brings together survey companies and charitable groups in a mutually beneficial relationship. The charities share their donor lists with the survey companies, who then tell those donors they will make contributions to their favorite charities if they participate in surveys.

Not only does the program remove the incentive to cheat, it gives marketers access to people who might normally avoid participating in market research. "There's a frustration among researchers that they're tapping into the same panels again and again," says Bryan DeRose, senior director of business development for the CMO Council. "A lot of these folks we're reaching through Pause to Support a Cause are a bit harder to reach, folks that may not normally take a survey but will if it's endorsed or will benefit a group they're used to supporting."

But even with the problem of incentives removed, online panels still suffer from the same fundamental problem dogging traditional focus groups: They don't measure what people think, they measure what people say when they are speaking to marketers.

The ability to collect data based on what people say when they're not being watched - or at least when they don't think they are - is what makes people like Pettit so excited about social network data mining.

To illustrate the purity of the opinions she commonly receives through her research, Pettit talks about a current project that involved gauging public opinion of a particular actress. People on the social networks she was monitoring "were more than vocal and specific about" their "amorous feelings," she said. "You would never get that language on a survey."

It's an extreme example, but it gets to the heart of social media research's potential.

Of course, plenty of researchers have their concerns. "The fact is that most people don't talk about brands online, and if they do, they talk about the stellar ones, like Nike and Snapple," said Poynter. "So if you're making canned tuna or something," there may not be enough conversations to yield significant data.

He also points out that what people talk about online is not necessarily what brands need to know. "Porsche might be interested in making some changes to their dashboard," he says, "but nobody is talking about how Porsche's dashboard could be different. . . . It's a bit like astronomers who have to wait for a supernova before they can study something, or who don't know an asteroid is about to hit us until it is on the way."

Pettit refutes these criticisms to a point - "I rarely come across a case where at least a few people aren't talking about whatever you're interested in," she says - but acknowledges that social network data mining has its limits. "We can never replace surveys or focus groups completely."

As long as there are clients who insist on seeing a consumer's body language, she says, or who need consumers to physically interact with a product to comment on it, in-person research will survive. "My theory is that every methodology has good points and bad points," she says. "All have certain objectives they can reach."

The real question, says Poynter, is how long any methodology can be effective before the people being monitored figure out they're being watched. Ultimately, researchers need to constantly be coming up with new methods, if only to stay one step ahead of the consumers. "It's kind of like an arms race," he says. "Customers are becoming increasingly savvy, so the people who measure them have to become savvier."