Gaming rating systems

[Kupek]

Perfect scores are inherently bogus. If they are to mean a game is absolutely perfect ...

I don't think the reviewers make this claim. A 10/10 doesn't mean the game is "perfect" - as in, impossible to improve upon - just that it represents the best of what's out right now.

Frankly, I think that using a number scoring system to rate games is broken. What does a 5/10 represent? Is that an "average" score, or a really low score? I've seen magazines and news sites try to get across that when they rate a game as 5/10, it's an average game. But I doubt that the people who review the games are so impartial that they don't inflate the scores a little bit - 5/10 just doesn't <i>seem</i> average.

If we're going to talk reasonabley about the relative quality of games, I think the best system to use is the grading system most of us are familiar with: A, B, C, D, F. We all know what they mean, so there's no need to further explain them. A is for the best games, for whatever reason. B is for games that are good, but have enough faults that the A games are noticabley better. C games are average, for whatever reason. D games can see mediocrity but have some failing severe enough to prevent them from achieving it, and an F game has little or no redeeming value. This gets around the silliness of giving a game a 93.5% or a 7.5- what does it even mean to be so precise with something so subjective?

I realize that we can achieve the same result with any scale that has five values, but we already know what A, B, C, D and F mean in terms of quality.

[Kupek]

Bwahahaha. I just realized that I said "we all know what they mean so there's no need to further explain them" and then I proceed to do just that. Anyway, my point is that it's a system we're familiar with.

[Don]

If 10/10 doesn't mean it's a perfect game then a 10/10 is meaningless. Like I say in that case all it means is that 10/10 or whatever the max is means 'good enough that the problems aren't a problem'. Whose standards do you use? If the reviewer doesn't have the same standards as you, is the score even meaningful? The point of using a fine-grain scale is supposed to provide some kind of objective evaluation of a game. For example if I think FF7 and FF8 are the two best games ever but FF7 is a bit better, then FF7 might be 10.0 and FF8 might be 9.9 and while the #s are arbitrary you'd expect FF7 to have a higher number than FF8 in any scale. Whereas if we've a 5 star or A/B/C/D/F system then we'd just get 5 stars or A for both games and you can't convey the fact that you believe FF7 is better than FF8.

Now of course people really should read what is written to get a feel of how good the game is as opposed to a numeric value but that's probably asking too much.

[Kupek]

If 10/10 doesn't mean it's a perfect game then a 10/10 is meaningless.

Have you ever gotten an A paper? Did that mean your paper was perfect? No, it just meant that at the level you were at, yours was the best of the best. (Numerical score on a test doesn't work here, since that's an objective measure, and there's no complimentary thing when it comes to evaluating a game.)

Like I say in that case all it means is that 10/10 or whatever the max is means 'good enough that the problems aren't a problem'.

And what's wrong with that? That seems to be the only reasonable interpretation to me.

Whose standards do you use?

Mine. Whose standards does your professor use? Whose standards does a movie critic use? But we all understand that it's a subjective judgement and take that into consideration.

If the reviewer doesn't have the same standards as you, is the score even meaningful?

Not really. But that's a problem inherent in reviewing anything; having a finer grain scoring system isn't going to solve that problem.

Whereas if we've a 5 star or A/B/C/D/F system then we'd just get 5 stars or A for both games and you can't convey the fact that you believe FF7 is better than FF8.

And why is that important <i>for a game review</i>? If the professor gets two A worthy papers but prefers one slightly over the other, is it important for the grade to reflect this? Not if they're both A worthy, I don't think.

The purpose of a game review is to evaluate a game so that others may have an idea of whether or not they'd like it, not establish a total ordering on all games in existance.

[Don]

Besides the fact that people are obsessed with ordering everything in existence (hence the top X things of all time stuff all the time), having an ordering of many games allow you to get a better indicator of the accuracy of review via collaborative filtering. For example if I agree game A is 9.0 and game B is 9.5 with reviewer A then I can reasonably assume if reviewer A reviewed game C as 9.2 there's a good chance I'll agree that this game is better than game A but not as good as game B. This even works even if I don't agree with the reviewer (for example, if I think A&B are messed up then the score would indicate that it's better than B but not as good as A). While by no means foolproof it's a lot more reliable than this game A is A and game B is A and game C is A which is the corresponding score they'd have on a 5 point system.

Further all gaming reviews are invariably postively biased for the reasons you mentioned: people inherently thinks 5/10 = bad. When you only have 5 numbers to work from the bias throws off the reliability even more. With 5 numbers everything invariably clumps toward the '4' range which, on a 1-10 scale, would typically encompass the higher 6s to the lower 9s. While a 1-10 scale would probably see scores biased toward 7s or 8s that covers a smaller area of uncertainty, i.e. if scores are biased towards 7s it can't represent more than high 6s to lower 8s.

[Kupek]

Besides the fact that people are obsessed with ordering everything in existence (hence the top X things of all time stuff all the time), having an ordering of many games allow you to get a better indicator of the accuracy of review via collaborative filtering.

That's still possible with an A, B, C, D, F rating system. The granularity you were talking about just isn't necessary to get a feel for how a reviewer thinks of games compared to your tastes.

When you only have 5 numbers to work from the bias throws off the reliability even more. With 5 numbers everything invariably clumps toward the '4' range which, on a 1-10 scale, would typically encompass the higher 6s to the lower 9s.

But what does it really <i>mean</i> for a game to get an 8 versus a 9? Or a 2 versus a 3? After a certain point, why bother saying how bad it is? Are you really able to get a feel for how that translate to what you might think about it? A, B, C, D, F gets across the meaning clearly. You can throw in pluses and minuses to achieve a higher granularity, even though I think it's silly.; what I think is important is the semantics of letter grades. We know immediately what a grade means.

[Nev]

and speaking of rating inflation...i dunno what it's like where you go to school, kup, but at stanford I think the average in most classes was more like a "B". my high school also. I'm guilty of it as well...on a 10/10 scale I usually look for 7 as "average", and if a game gets a "C" in a review mag it's a tough sell trying to convince me to play it. I suspect that our country as a whole may be grade-inflated such that a "B" is considered more of the average these days.

[SineSwiper]

and speaking of rating inflation...i dunno what it's like where you go to school, kup, but at stanford I think the average in most classes was more like a "B". my high school also. I'm guilty of it as well...on a 10/10 scale I usually look for 7 as "average", and if a game gets a "C" in a review mag it's a tough sell trying to convince me to play it. I suspect that our country as a whole may be grade-inflated such that a "B" is considered more of the average these days.

Blame the US's grade-school explaination of Communism.

[Don]

I remember hearing grade inflation started from World Wars because they don't want you to feel bad about your grade if you're about to go to a war and die.

[Don]

What makes a 9 better than an 8? Maybe for any 2 particular game I can't give you a good reason, but if I rated 10 games 9 and 10 games 8, it should not be an accident. Every game I rated a 9 ought to be, no matter how inexplicable, better than every game I rated a 8 assuming I'm reasonably consistent. If reviews are meant to be some kind of guide as a predictor of game quality, then they better have some kind of ordering to them. If I have 20 games rated A and the next game is rated A that doesn't tell me as much as 10 games rated a 9 and 10 games rated an 8, and the next game being a 9. Now at some point the scales probably do become meaningless (e.g. 9.7 versus 9.6) but I think 5 is too coarse.

[Zeus]

If 10/10 doesn't mean it's a perfect game then a 10/10 is meaningless.

I disagree with that. 10/10 is more like "better than we thought possible at this current stage of gaming for that genre with negligible or no flaws". I mean, FF6 back in 1993 is an easy 10 but nowadays it's not gonna be close. All games have to be judged for that time period and that's what the 10 represent. Best for that time period, which includes comparisons to other games of the same genre, the level of gaming at that time, and the power of the systems.

[Don]

That definiton is totally meaningless. The point of having numbers often down to unrealistic fine-grainness (i.e. 10.0) is to infer some kind of objective evaluation. What you're saying is that 10/10 would be the equivalent of 'best game ever' which is hardly objective at all. All such a rating just means that it's right if it's convenient to agree with them. Like I mentioned Vagrant Story is a very good example because there is no way you can say the flaws are negigible. They do impact the gameplay and while it is a good game you're simply not allowed to handwave and say it's okay, it's still 'perfect' when people look at you as some kind of definitive source of gaming wisdom.

[Kupek]

If I have 20 games rated A and the next game is rated A that doesn't tell me as much as 10 games rated a 9 and 10 games rated an 8, and the next game being a 9.

The major difficulty with number scoring system is that I don't know what an 8 <i>means</i>. You can achieve the same granularity using pluses and minuses, i.e., 10 games rated A-, 10 games rated B+ and the next game A-. I'd rather avoid pluses and minuses, but I still think that's an improvement over a system that uses numbers.

The point of having numbers often down to unrealistic fine-grainness (i.e. 10.0) is to infer some kind of objective evaluation.

Reviewing games (or books, movies or music) is inherently subjective, so I don't see how having a fine grain scoring system can achieve any kind of objectivity.

[Julius Seeker]

It's actually quite simple. When a reviewer rates something a 10, it means that whatever category the 10/10 is rated under could not be any better in the eyes of the reviewer. Different reviewers have different objectives in mind, similar to the four reviewers of Famitsu who each scored Nintendogs a 10/10. The reviewers of famitsu are also known to be among the toughest reviewers in the world.