Our Rating System

The 5-point rating system explained

We feel that many products do not need so much granularity when summarizing a product. The main reason we review products is to inform our readers, and a 5-point rating system is simpler and more effective.

In the early days, OCModShop used to have a subjective 10-point rating system, and we found that horrible products were rated higher than they really deserved. One of the worst products I’ve ever reviewed was the Waffer Aircon PC air conditioner, which originally received a subjective 6 out of 10 (because I was trying to be nice and figured that making a product was worth a few points). This should have received a 1 out of 5 to truly identify the waste-of-plastic that it is.

Case study: Several review sites gave PS3′s Mobile Suit Gundam (the worst piece of shovelware on any system) a 3.1 to 3.9 out of 10 points. Do you really need to know that a game is a 3.2 of 10 or 38% to know it should be avoided like warm fish off a truck? Using the 1 out of 5 scale is a simpler way of stating that this game is terrible.

Some may argue that really great products aren’t given their credit, saying that it’s too easy to achieve a 5 out of 5 rating. This is why we rate the product in several categories, and it really isn’t that easy to achieve perfection…

Rating Products Objectively
Different reviewers value different things, and allowing one reviewer to pull a number out of their butt isn’t fair to the product or product manufacturers.

To add objectivity to the final score, we rate the product in 5 categories and average that total. Instead of assigning an arbitrary and subjective overall score, we let the numbers and categories decide the rating. The average score usually ends up as a decimal, and is rounded up or down to the nearest whole number.

Product Rating Categories
Game Categories Hardware Categories
  • Creativity – Does the game breathe new life into the genre or approach an idea in a creative way?
  • Sound – How realistic are the sound effects? What about sound positioning? Soundtrack quality?
  • Longevity – How much gameplay are you going to get out of this game?
  • Graphics – How do the graphics compare with other games on the same system? Does it make innovative use of the hardware? What about texture quality?
  • Fun Factor – All else being said, is it fun to play?
  • Quality – Build quality of the product. Will it break in 3 months or will it last a lifetime?
  • Innovation – Does the manufacturer try something new, or stick with the tried-and-true?
  • Performance – How well does it work? Does it perform as advertised?
  • Installation – How much effort does it take to get this working?
  • Value – Are you getting your money’s worth? Some products may be extremely expensive, but may be a good value depending on the quality, or included extras.

What goes around…
Since we use a 5 point rating system, some may think that we rate in 20% chunks. Some may think that any score above 80% automatically recieves a perfect score, and this is simply not true. We add up the scores in each category and average them to come up with a final decimal number, which is then rounded up or down to the nearest whole number.

We round up to the nearest whole number because if we used decimal points, then we might as well be using a 10-point or 100-point system, which defeats the simple nature of our rating system. A 4.5 out of 5 easily translates to 9 out of 10. A 8.1 out of 10 is harder to remember and translates to 81% of a 100-point scale.

Do you really need that much granularity?
Fanboys of a particular genre will say yes, because they love to boast that Madden 07 is better than Madden 06 because it has a 2% lead over the previous generation. The point is: it is the same frickin’ game!

Did your parents care if you got a 93 or 95 on your exam, or that you scored an “A”? The only people who cared were the two nerds fighting it out between who was the best “co-valadictorian”.

Highly granular rating systems can become obfuscating that you must examine every word of the review to find the nuances that really matter to you. Some games may not score as well because they have lower-quality sound, while others may receive the same score because they have a bad aiming system, but everything else is excellent (e.g. Dead Rising). Again, this is why we break the products down into several categories: everything is in a small easy-to-digest fun-size chunk of creamy goodness.

1 – Terrible
Should be avoided, even if it’s free.
2 – Bad
You probably won’t like this product, so save your money. Might be worth it if it’s in the $5 bin.
3 – OK
It delivers exactly what you expect: no more, no less. Diehard fanboys of a particular brand will write flaming emails on why we don’t see it their way.
4 – Very Good
There are a few flaws with the product, but overall is very good.
5 – Excellent
There are very few flaws. nearly everyone will like this product.
Perfect
Receives a 5 in every category. We cannot find any faults with this product.

The Critics
Critics to our system will say that some products receive a 5 out of 5 when there are clearly some flaws. This is true: Some flaws, but not very many. Even if we demoted a 4.5 (which rounds up to a 5) to a 4.4 (which rounds down to a 4), it is still a good product. In the rare case that every category receives 5 points, we have reserved the “perfect” badge.

It’s all about informing you, the reader, and we feel our 5-point rating system weighs all the pros and cons fairly and delivers an easy-to-remember rating. It all comes down to recommending a product, and the bottom line is if we would recommend it or not. Let’s not scrutinize over it…

Feel free to leave comments below and let us know what you think!

0 Comments

You can be the first one to leave a comment.

Leave a Comment