Can We Trust Internet Ratings?

Bob Thompson

Can we trust "expert" ratings, either?

Chris, you raise a great point about the ratings. However, it's well known that surveys of any kind tend to skew towards the high end of a scale of 1 to 5 , 1 to 10, etc.

Here's a post from Vovici (survey company) that does a pretty good job of explaining:

Years of data show that the spread of a five-point scale usually skews heavily towards the top two answer options. Thus, when using five-point scales, we'll find that:
* 88% of respondents answer 4 or 5
* 10% answer 3
* 2% answer 1 or 2

There are techniques to deal with this, but clearly a simple Internet rating on Amazon.com or elsewhere should not be trusted in the sense of taking the score at face value.

I do agree with John that the comments are more valuable. When buying a piece of electronics recently, the average scores did catch my eye, but I didn't give it much weight unless there were a lot of ratings. The comments about experiences, pros and cons, etc. were very helpful in my eventual decision.

Whether valid or not, users tend to trust other user reviews. Here are some interesting stats on the power of word of mouth:
http://www.bazaarvoice.com/industryStats.html

The thing is, is the alternative any better? Editorial or analyst reviews also skew to the high side. Look at CNET editorial reviews and they look a lot like the user reviews, sometimes even more positive.

And what about software analysts in our own industry? It's amazing to me that so many vendors are considered "leaders" in one form or another. And these are ratings built on complex processes and spreadsheets.

The current economic crisis was helped along in large part because the rating agencies (Moody's etc.) gave AAA ratings to those clever packages of subprime mortgages, which then were bought up by financial institutions worldwide who thought they were buying "quality" based on the rating.

So, I'd be hard pressed to find any source where ratings are distributed in a bell curve. Personally, I wouldn't put my trust totally in user or expert ratings/reviews -- it's the combination of the two that gives a better picture of reality.

Bob Thompson, CustomerThink Corp.
Blog: Unconventional Wisdom

Graham Hill

I'm With Bob on This One

Chris

As Bob points out, surveys, particularly satisfaction surveys, have always suffered from positive response bias. There are ways to avoid this in survey deisign, e.g. by having a combination of positive and negative response questions, and by using advanced statistical methods to control for the bias during analysis, but they are for professional statisticians.

Mere mortals should take advice from a variety of on-line and off-line sources, and handle the product themselves before buying. When swithing from a semi-professional analogue SLR camera to a digital one, I read all the on-line reviews, talked to fellow photo-enthusiasts and went to a camera shop to handle the three cameras which were possibilities. The camera I eventually bought was the top choice of most of the reference sources and was my top pick in handling too. I bought it for the lowest price from Amazon.

Graham Hill
Customer-driven Innovator
Follow me on Twitter

Interested in Customer Driven Innovation? Join the Customer Driven Innovation groups on LinkedIn or Facebook to learn more.