• by Douglas Quenqua , April 13, 2011

It's true: Advertisers lie. But here's another fact: so do consumers. Anyone who's ever monitored a focus group can tell you that. Maybe it's not lying of the shameless "It was like that when I got here" variety. It's more the "I base my grocery-buying decisions on nutrition, not packaging" kind of fibs. But either way, it's misleading and advertisers pay the price for it.

Luckily, faces don't lie. Someone watching an offensive commercial is going to look disgusted, just as the lips of someone watching a poignant ad will quiver as he fights back tears. All the questionnaires in the world can't match what a trained professional (or now, software) can learn from watching someone's face. If only there were a way to harvest the facial expressions of consumers of all demographics from all over the world, then translate them into actionable marketing data.

To be clear, there isn't...yet. But there are a handful of companies doing some clever things with facial coding and eye tracking and other whizbang technologies to help advertisers better gauge how their customers really feel about them. Or at least how they feel about their products, packaging, ads, store designs and Web sites. "Darwin realized that the face was the best place in the body where we reflect and communicate our emotions," says Dan Hill, president and founder of Sensory Logic, a market-research firm based in Minneapolis, Minn. "It's the only place in the body where muscles attach right to the skin." Darwin's insight was based on the observation that blind people make the same facial expressions, corresponding to the same emotions, as sighted people, suggesting that facial expressions are not learned but hardwired into our brains. Facial coding may owe a philosophical debt to Darwin. But the technology traces back to a psychologist named Paul Ekman, who in the late 1970s developed what's known as the Facial Access Coding System, or FACS. The system, which has so far been of greater use to animators than advertisers, reduces all facial movements to mathematical units. By measuring the movements of a person's face, one can interpret that person's emotions.

Thirty years after the development of FACS, companies like Sensory Logic are helping advertisers apply that technology to market research. The primary way Sensory Logic does this is exposing test subjects to a commercial or a radio ad while a Logitech camera records the subject's face (the process is called "exposure"). Experts then decode the facial expressions and line them up with the content the subject was watching to determine, down to the second, how that person really felt about what he was watching. In the past few years, the company has even begun offering this service remotely. As long as a subject has a camera installed in his home computer, the company can record his facial expressions as he watches the content from home. Other companies are now working on bringing the technology to smartphones.

The invention of the Flip camera has also allowed Sensory Logic to combine facial coding with traditional focus groups. Now market researchers can record the faces of focus-group subjects to see when their expressions betray their words. The company can also record the faces of subjects answering questionnaires to look for "any gaps or illumination between what they see and feel," says Hill. To appreciate the value of such tests, consider the example of a pharmaceutical company trying to determine what mattered most to doctors when prescribing medicines. When asked why they choose one drug over another, "of course the first thing the doctors say is efficacy of the drug," says Hill, "but when we looked at the emotional data they hardly emoted about that at all." What did capture the doctors' hearts, according to their faces? "Ease of filing for reimbursement and the payment schedules and so forth," says Hill. In other words, "things that were much more central to their own pocketbook. Those things were much more of interest to the doctors."

This kind of research has also helped quantify the shortcomings of traditional focus groups. Most advertisers already know that people in a focus-group setting will sometimes adjust their answers to make themselves look smart, or to cozy up to an alpha dog in the room. But Sensory Logic has taken this further by looking at how people of different ethnicities and demographic backgrounds behave differently. "Hispanics tend to respect authority, which means they might be a little less forthcoming to make a criticism," Hill says. "African-Americans are pretty straightforward. In our research they come closest to just saying how they feel. But whites and Hispanics and Asian-Americans - you're talking about a gap that's double digits between how much they're willing to admit they like something" and how much they really do. In an age when advertisers spend a lot of time and money crafting discrete messages for different ethnic groups, such lapses in insight can add up to significant waste. Sensory Logic is also eager to take facial coding outside the research room and is looking to partner with companies that provide eye-tracking technology. Such companies - EyeTracking, located in San Diego, and Tobii, in Sweden, are two - use special glasses worn by the consumer to determine where his eyes settle as he walks through a retail environment. Combine that with facial-coding technology and you can tell not just where people are looking, but how they feel about what they're seeing. The application has exciting implications for retailers, who could use it when designing their store layouts. Of course, it wouldn't be an emerging technology if there weren't disagreements over how best to apply it. Gallup & Robinson, a market research firm in Pennington, N.J., practices a different brand of consumer face reading.

"Our focus is on what we call emotional valence," says G&R president Scott Purvis. "Most of the other systems are on what we would call arousal."

To the layperson, the difference between what Sensory Logic and Gallup & Robinson do may seem tiny. But it's significant. Put simply, emotional valence is the measurement of all human reactions on a scale of negative to positive. So Gallup & Robinson is less concerned with whether you're smiling or frowning than how intensely you're doing one or the other.

"The idea is that people have a positive and a negative rewards-type system in their brain," says Purvis, "and these things operate both at the same time, so you can feel both very positive and negative toward a stimulus at the same time. It's not ends of a scale, it's two different systems, so that's why people are often ambivalent about many of the things they're seeing. We're trying to get at that relationship, rather than whether somebody is disgusted or happy." Purvis uses the example of a typical pharmaceutical commercial, the kind that begins with patients suffering from an ailment, then features a (frequently animated) illustration of the treatment working, followed by scenes of a healthy, happy patient. The intensity with which a viewer follows those phases of emotion "can be very important in terms of overall effectiveness of the commercial," he says. "And I don't know that we understand it as well when looking at it from the point of view of, 'Did you like it?'"

Not surprisingly, Hill doesn't put much stock in emotional valence. He points out that there is a wide range of emotions that can be considered "negative," not all of which are the ones you may want your commercial to inspire.

"Sadness versus anger versus fear are all quote-unquote negative emotions," he says. "But those emotions can not only be vastly different in their meanings, but also in their implications for your creative."

Hill, too, uses the example of a pharmaceutical commercial to illustrate his point. "You're trying to create a problem-solution commercial, so you're supposed to make your viewer feel sadness at first because they're disappointed with the status quo," he says. "But what if you're actually making them angry because they're confused? Then you've lost them."

Hill even has a term for it. He calls it being "off-emotion," as in being off-message.

Measuring emotional valence comes with some other trade-offs. Rather than making video of a subject, Gallup & Robinson attaches electrodes to his or her face, specifically at the frown and smile muscles. The process, known as electromyography, or EMG, may do a more accurate job of measuring the intensity of muscular reaction - but it doesn't travel well.

"There's some work being done now to use the camera to distribute it over the Internet so Webcams would be able to get an EMG-type response," says Purvis. "But so far it hasn't gotten to the point where I think people have a lot of confidence in it."

Regardless of which method you put your faith in, there's a good reason that both remain little more than a theoretical threat to traditional market research: They have a serious scalability problem. Both approaches require trained professionals to translate the results. Training such professionals is said to take up to 100 hours; of course the process of watching the videos and drawing conclusions takes many hours more. So while face reading makes a nice complement to focus groups or questionnaires, they've never really been scalable enough to replace them.

Enter ThirdSight. This 11-person firm in the Netherlands may be the newest thing in face-reading market research, having been spun off from the University of Amsterdam on January 1, 2011. ThirdSight's signature technology is software that reads and interprets video of test subjects' facial expressions, eliminating the need for a pricey, highly trained researcher to wade through hours of slow-motion video.

"The software detects the face of the subject, then localizes landmarks like mouth corners, cheeks and eyebrows and tracks these facial features over time with a grid like a spire," says Theo Gevers, scientific advisor at ThirdSight and an associate professor of computer science at the University of Amsterdam. The software is taught to recognize facial configurations that correspond to six different emotions, so it can tell instantly how a test subject is reacting to content.

ThirdSight has even created a smartphone app that can read and interpret a user's facial expressions. "It's not yet real time," says Gevers, "so it will record your face and then process it." But that will change quickly as smartphone CPUs become faster and more powerful.

The drawback to ThirdSight's process is an admitted lack of accuracy. "We might have some disadvantages in the sense that people who have been trained for 100 hours might be a little bit better" than the software is at reading the subtleties of facial expressions, Gevers notes. "We have a fairly accurate system, but it's not as good as a human operator."

Still, the potential effect of scalable facial coding on the world of market research is huge. Imagine a world in which consumers anywhere could download an iPhone app that records their faces as they watch a commercial - which also plays on their phones, incidentally - then instantly beams back that person's emotions to a waiting database. Today, companies like Sensory Logic and Gallup & Robinson are limited to running sessions with about 40 people; more than that would simply be untenable given the man-hours that go into interpreting the results. But a smartphone app equipped with software that can read faces in real-time has the potential to make facial coding an easy, inexpensive mass-market research tool with a higher level of reliability than anything available today.

"The conservative estimation is that 95 percent of people's thought activity isn't fully conscious - that we're mostly intuitive, subconscious decision makers," says Hill. That, more than any other reason, is why simply asking consumers how they feel about a product or campaign is bound to come up short. No matter how much they want to tell the truth, people just might not be able to. Bypassing the consumer to go directly to his brain, courtesy of his expressions, may be the market research industry's best chance to save face.