Fuzzy Insight: Social media driving interest in sentiment analysis

At a local meetup organized by Tatyana Kanzaveli, Esteban Kolsky did his usual masterful job moderating a panel of "sentiment analysis" experts from Attensity, Bing, and Scout Labs.

A few highlights from a excellent discussion...

For those not familiar with the term, sentiment analysis means the use of text mining/analytics to help determine whether written text has a positive or negative tone (hence, sentiment) and why the writer was happy or mad.

Text mining has a long history and lot of complicated algorithms at the core, but in recent years the explosion of social media has ramped up interest. The massive amount of content generated by users on Twitter, blogs, forums, etc. is an excellent opportunity to gain insight, learn about developing problems before they show up on CNN and improve the user experience.

So for example:

• Attensity can help companies enhance Voice of Customer programs by analyzing write-in comments and other text feedback.
• Bing can help users find the best products by analyzing reviews posted on the Web.
• Scott Labs can help monitor the health of a brand by analyzing Tweets and other social sources

But text analysis is not an exact science. Well-trained people might achieve a 90% success rate in categorizing text documents. Good automated methods might hit 70-80%, but it all depends on the task, how the application is trained and tuned, and the methodology being used.

And some text analysis problems are nearly impossible to solve, like understanding sarcasm and metaphors. I guess us humans will continue to have some work to do for a time...

In short, you won't get the precision that you're used to with real data, but text analysis is the only effective way to deal with huge volumes. Given the choice between fuzzy insight and no insight, most business leaders will choose the former.

I've been following text analytics the past couple of years because it's a cost-effective way to gain insight from unstructured data. Now social media has made this challenge and opportunity even more compelling.

On a industry note, I think text mining in various forms will become deeply integrated into mainstream CRM applications like customer service, while also continuing to be packaged into focused SaaS-based applications for VoC, social media monitoring and more. The next couple of years should be very interesting to watch as vendors scramble to incorporate and package the technology.