Relying On Sentiment Analysis To Predict Content Impact? You're Making A Big Mistake

by Fard Johnmar, Feb 15, 2013, 9:03 AM

• Comment
• Recommend (1)
• Tweet
• 

Recently, I came across an interesting article published in the pharmaceutical marketing publication PM360. It features an examination of the online reputations of the top 10 U.S. prescription medicines as ranked by direct-to-consumer advertising spend. A fundamental component of Michelle Bennett’s (who serves as COO of Wool Labs) analysis is an assumption that negative online content and conversations can negatively influence customer perceptions and potentially behaviors.

Is this assumption correct? For many years a range of companies and organizations have used sentiment (whether content is positive, negative or neutral) as a proxy measure for content impact. I was a believer as well until I came across research published in 2008 by the social network PatientsLikeMe (PLM). In the study, PLM looked at how muscular dystrophy patients reacted to widespread news that the medication Tysabri was associated with new cases of progressive multifocal leukoencephalopathy, or PML. PLM found that safety concerns dominated conversation following the PML announcement but it didn’t lead to a negative perception of the brand. Instead, patients thought the benefits of the medication outweighed the risks.

Investigating whether there is a link between sentiment, perceptions and health behaviors is a major goal of digihealth pulse. In our studies, we ask participants to rate the sentiment of content in addition to telling us whether it shifted their perceptions or health behaviors. To date (in studies conducted between 2011 and 2012), we’ve collected thousands of responses to questions about how content influences perceptions and behaviors across more than 40 health topics. Analysis of this large data set has led us to the conclusion that sentiment is at best an imperfect predictor of content impact.

Overall, we’ve found:

• Sometimes positive or negative content can activate/deactivate health behaviors, but other times it does not
  
  
• Neutral content is the black box of sentiment analysis, as computer algorithms often automatically rank content as neutral; however, surprisingly, we’ve found that neutral content sometimes sparks positive health behaviors

In order to explain these concepts further, we’ve developed a visual report that contains sample data from studies we’ve conducted with online moms and health providers in the area of HPV. (This information is representative of the data we’ve collected thus far.) We wanted to determine whether HPV-related content prompted:

• Providers to recommend or prescribe HPV vaccinations
  
  
• Mothers to ask doctors about the HPV vaccine or pap smears

We found that sentiment was not always linked to how mothers and providers would act. Instead, their reaction to content was dependent on their status (e.g., whether they had children), the context of the information provided and other factors. In addition, neutral content published by government organizations sometimes played a role in sparking intent to engage in a behavior.

Sentiment is an attractive metric because it is easy to understand and report. However, we should be careful about over-relying on sentiment as a measure of content impact. 

To download the visual report, please click here.

• Comment
• Recommend (1)
• Tweet
• 

Be the first to comment on "Relying On Sentiment Analysis To Predict Content Impact? You're Making A Big Mistake"

Leave a Comment