Home > economics, information technologies > Shortcomings of Rating

Shortcomings of Rating

In recent years one could observe a mass proliferation of all kinds of ratings and applying their results to many activities in our life, often without full understanding how rating are collected and what shortcomings they have. I do not pretend to cover all problems with ratings, but will describe a few.

First of all, when I say ratings I imply a request to express a personal opinion on some matter. This leaves, for example, credit ratings out of this topic. Ratings I am referring to aim at collecting personal preferences. Ratings come in many different flavors. For example, in the Internet ratings come from “Like” in Facebook, to “Thumb up” and “Thumb down” in YouTube, stars, X’s, numbers and so on. In real life, you could see, for example, a request to fill out surveys on a customer service call or just fill out some information as part of a profile creation process. Now we can talk about shortcomings of ratings.

  1. Context often is not captured. This includes things like your location, people you are with, your feelings, etc. Using this rating to “help” you will result in inappropriate advices. For example, while visiting Arizona you ranked a good ice-cream shop where one can get a good chill, but now you visit Alaska and want to find a place to eat and stay warm, and the system that studies your preferences now suggests a good ice-creamery just around the corner. The take-away from here is ratings are not solely linked to the subject being ranked. Whoever finds out a way to capture as many parameters as possible during the rating process provided that the other two problems with ratings are solved (where applicable), will be golden.
  2. Effort is not taken into account. This is my favorite. A typical example. You call technical service and it takes 1 minute to solve your problem. The next thing the technical service does after your call is it sends a survey “How did we handle your support call?” that requires 15 minutes to fill out. As you can imagine, not many people would respond to those surveys. In most cases, technical support in these cases will receive very few responses “Met expectations”, if any.
  3. Incentives to cheat. This one is difficult, but an example will help. Management of the building where I lease an apartment every once in a while sends a survey “How do you like living here?”. The problem is this is a large company that uses every bit of information to maximize its profits. So, if I respond “I like it”, next time when time comes to extend my lease, I may see a higher number in my new lease than, for example, in case if I reply “I hate it and plan to move out”. This is also known as perverse incentive in economics.

Because of these problems, ratings/feedback collected in many cases is useless and do not show the true picture. What other problems with ratings did I miss?

  1. No comments yet.
  1. No trackbacks yet.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: