Review Score Inflation Charts

[Kefkataran]

Noticed this mentioned over at Joystiq (http://www.joystiq.com): A new gaming blog, MetaFuture (http://www.metafuture.com/) , is positioning itself as a review site watchdog. Their first task? Charting review scores on big-name gaming sites IGN and Gamespot. From the Joystiq post (http://www.joystiq.com/2006/08/07/ign-gamespot-review-score-inflation-revealed/):

IGN's average review score was a whopping 8.0, and the site heavily favors evenly rounded scores. Gamespot, while a little more balanced and even in their curve, has an average review of 7.0. So either there are a butt load of great games we're overlooking, or something fishy is going on.

Metafuture sounds like a great site to keep an eye on. If they drum up enough attention, there could actually finally be that much-needed push to make 5.0 = average like it should.

[Ventura_DK]

Hmmmm Gamerankings have been around for ages - easy to see how the sites rate on average, even compared to the big overall average rating of all sites combined.

Apart from the big sites, many smaller sites suffer from the fact that publishers never send them the really crappy games anyway. So if you only get good or semi-good games, you can't get your average review score below 7-ish.

[Kefkataran]

Apart from the big sites, many smaller sites suffer from the fact that publishers never send them the really crappy games anyway. So if you only get good or semi-good games, you can't get your average review score below 7-ish.

That's completely true, but we're talking about the big sites here specifically.

[Mozgus]

School grading structures do not apply here. 5.0 should be the god damn average.

[random johnald?]

I think the problem is that, for the smaller sites at least, companies will quit sending them games if they realize that they rank them realistically.

A company that makes an average game would rather send an early preview to IGN for a 7.5 rating than to some smaller site for a 5.5 rating.

[thecrazyd]

This is why I recommend, at most, a five point review score with no decimals. With 100 point reviews it is easy to issue arbitrary values to different aspects of the game resulting in skewed score values. With a five point system, it comes down to pretty much how much do you recommend this game. It also encourages the reading of the actual reviews. So, I guess I would not recommend this system to IGN as their reviews are horrible, never ending works of torture.

[Kefkataran]

School grading structures do not apply here. 5.0 should be the god damn average.

Agreed.


A company that makes an average game would rather send an early preview to IGN for a 7.5 rating than to some smaller site for a 5.5 rating.

And again, this is why Metafuture is looking at IGN and Gamespot and not smaller review sites (as of yet).

This is why I recommend, at most, a five point review score with no decimals.

I totally, one-hundred percent agree. I always prefer a four- or five-star rating system to anything else.

[Ventura_DK]

That's completely true, but we're talking about the big sites here specifically.

I know, I just don't think this new site brings much information that you can't already get out of Gamerankings.

However, I do agree that the high average scores are a problem. I think it's a bad combination of reviewers being either too kind or too inexperienced (having reviewed over 300 games myself I know how long it takes to get enough reference in any genre to rate correctly) and publishers not wanting bad games being reviewed.

[soco]

i think the school structure isn't really that bad here. there are so few games that truly stand out, reserving some 40 places to describe how awesome they are seems a bit extreme. there are however varying ranges of complete crap games. "seems like an awful waste of space"

[Heretic Machine]

Who says 5.0 isn't already the average? I'm betting if I were to grab every piece of gaming software in a game store, at least 6/10 of them will outright suck.

[kokyunage]

School grading structures do not apply here. 5.0 should be the god damn average.

Wrong. Like it or not a majority of games are targeted at school age childreen. Any kid under the age of 18 pretty much drives the market. Kids get engrained the idea that a C grade (70%) is average, 80% is good, and 90% excellent. A kid would never buy a game that was 50% average even though mathmatically it makes sense. If no kid would buy the game, no publisher would advertise in a magazine/website that produces average scores of 50%.

Hell, even Evil Avatar In-House reviews produce an average score of 70% (3.5 eyes):

Jaws 2/5
NHL 3/5
Fifa 4/5
DOD4 4.5/5
Fear 5/5

Dead or Alive 4 got 4.5 eyes? Almost comparible to Fear? LOL. Anyway, my point is that every scoring system is flawed. Just read the text review.

[bapenguin]

This is why I recommend, at most, a five point review score with no decimals. With 100 point reviews it is easy to issue arbitrary values to different aspects of the game resulting in skewed score values. With a five point system, it comes down to pretty much how much do you recommend this game. It also encourages the reading of the actual reviews. So, I guess I would not recommend this system to IGN as their reviews are horrible, never ending works of torture.

That's why we do that here.

The school grading thing is a good point. Maybe it's so ingrained in our minds that 50% is pretty much the lowest score that many of the sites scale it between 50-100 so average games always tend to get 70-80%.

[protocol_image]

Dead or Alive 4 god 4.5 eyes? Almost comparible to Fear? LOL.

Comparible? No, not even close. The score represents the overall enjoyment of the game. And I would say that I've had an awesome time, but not perfect, with DOA4. Honestly, I wouldn't give FEAR a 5. It was really good....great mood/graphics/scripted moments; but not perfect by any means.

As for the rating systems, I do like the idea of a 5 point system. However, it then comes down to the moments when a game should be a 2, but gets a 3, just because 2/5 looks so empty. Just watch X-Play. They will verbally trash a game, but then give it a 3/5, when it sure sounded like a 1/5 or 2/5.

The fact is that there will never be a perfect solution....for any meadia. Hell, just look at how rigged the Oscars are. :)

[EvilBob46]

Despite of the inflated review scores, you can still separate the merely "good" from the great, and that's what I'm primarily looking for. GameSpot in general seems to shy away from giving out scores in the 9.x range these days, which is good.

[AspectVoid]

Honestly? I'd like to see sites just completely get rid of the review 'score' for games. If someone is interested in a game, they can read the review. That will get them a FAR better idea of whether the game is for them or not, as oppossed to 3/5 stars or a 65%.

A number of my all time favorite games have gotten low review scores when they were released. Suikoden 2? A 76 at Gamespot. Lunar? 75 at IGN. Frankly, review scores don't really do much good when it comes to finding out if a game is great or not.

[Stormwatcher]

I think that the new site is interesting... Metacritic, Gamerankings and Gamestats all just collect numbers, the new site intends to perform analysis on those numbers. That's really cool.

And the PCGamers of yore (1995-2000, US and UK) did have much stricter grading standards. A 9.0 was a sight to behold.

[DeadScreenSky]

Dead or Alive 4 got 4.5 eyes? Almost comparible to Fear? LOL. Anyway, my point is that every scoring system is flawed. Just read the text review.
That's true - DOA4 is much, much better than FEAR.

But it's not like EvilAvatar reviews every game or even a majority - clearly a huge number of lesser games are ignored, so you don't really get an average with any true significance.

[Serapth]

So... what you guys are trying to say is the average game can't be above average?

Thats just crazy talk!

[Spigot]

This is a great idea. I've been trying to figure out how to write an article about this for Toybane since about, oh, March. I'm sick of seeing good, but not GREAT, games being shat upon because they happen to get a score in the high 60's or low 70's. When I see people state that they won't touch a game rated less than 75% (whatever that translates into on the scale you're using) I have to feel sorry for them because they'll miss out on a lot of hidden gems and new concepts.

Granted, I like to read multiple reviews on Gamerankings to see what the sticking point is for reviewers. If I see a common problem that is a gamebreaker for me, then I'll stay away. If a game has great single-player but the multiplayer sucks and that is the sticking point, I'll ignore the ratings and pick it up as I'm more of a SP kind of guy.

Same with complaints about graphics. My beloved Chibi-Robo was bashed repeatedly by many of the review sites for having sub-par graphics. While it wasn't pushing a million polygons a second, I found the graphics to be quite charming and that they fit the style of the game. Yet the reviews for this hidden gem were consistantly mediocre at best. Every review said that it was off-beat, interesting and something different but because it didn't look like God of War or Halo 3, almost every reviewer gave it a poor score.

Sigh.

*end rant*

[SexualChoc]

You will never get a fair score for games reviews. I mean not everyone likes Game X, and it always has some flaws. Whether it's out of 5, 10, 40 or a 100. You should really just list the pro's and con's of a game and leave it at that. Also, it's IGN people, of course it's going to have a huge average. It's what they do.

[DeadScreenSky]

This is a great idea. I've been trying to figure out how to write an article about this for Toybane since about, oh, March. I'm sick of seeing good, but not GREAT, games being shat upon because they happen to get a score in the high 60's or low 70's. When I see people state that they won't touch a game rated less than 75% (whatever that translates into on the scale you're using) I have to feel sorry for them because they'll miss out on a lot of hidden gems and new concepts.
This is the absolute truth. And in my experience a person's reaction to a videogame tends to be far more individualistic than their average reaction to a book, film, or piece of music. It's for this reason and others that the "buying advice" angle taken by most professional videogame reviews is worthless.

[Shifteh]

Is this really a surprise to anyone? These are the same people who will give Doom 3/Half-Life 2/Halo 2/fucking NHL 06 all 5/5 or 92/100 or whatever they do. When I read PC gamers 98/100 for Half-life 2, I pretty much stopped using anything online or in magazines to help me decide on a game.

[Kefkataran]

I know, I just don't think this new site brings much information that you can't already get out of Gamerankings.

But it's not a site that gathers reviews the way Gamerankings does. Instead they calculate information about reviews and analyze that info. I don't really see the Gamerankings comparison at all.

Honestly? I'd like to see sites just completely get rid of the review 'score' for games. If someone is interested in a game, they can read the review. That will get them a FAR better idea of whether the game is for them or not, as oppossed to 3/5 stars or a 65%.

CGW did that. Or, well, Games for Windows now. It was actually a really smart, well-done change, IMO.

When I read PC gamers 98/100 for Half-life 2, I pretty much stopped using anything online or in magazines to help me decide on a game.

Just saying that maybe a well-loved game like HL2 (which, on a scale of 1-100, I personally would easily rate in the higher 90s) isn't the best example of their scores being too high. Doom 3's a better one, certainly.

[Rook34]

What if we got rid of scores altogether?

Imagine for a minute, a well thought review where instead of rating, it simply lists pro's and cons, perhaps outlining them in a seperate segment. At the end of the review the reviewer either says he liked it, didn't like it, or if the target demographic would. Then the reader is armed with what the reveiwer thinks personally, the pros and cons, and is able to make up their own mind.

Just a thought.

[Kefkataran]

Imagine for a minute, a well thought review where instead of rating, it simply lists pro's and cons, perhaps outlining them in a seperate segment. At the end of the review the reviewer either says he liked it, didn't like it, or if the target demographic would. Then the reader is armed with what the reveiwer thinks personally, the pros and cons, and is able to make up their own mind.

Again, try picking up an issue of CGW/GFW.

[Rook34]

Again, try picking up an issue of CGW/GFW.

I know CGW, but what is GFW?

It seems like it would be in the best interest of everyone involved to get rid of scores altogether, but then reviewers wouldn't be able to line their pockets from inflating scores!

[Shifteh]

Just saying that maybe a well-loved game like HL2 (which, on a scale of 1-100, I personally would easily rate in the higher 90s) isn't the best example of their scores being too high. Doom 3's a better one, certainly.

If you give a ranking like 98%, there had better be nothing, and I mean nothing wrong with that game. Game on rails? Check. Mediocre story? Check. Terrible teammate AI? Check. Mediocre enemy AI? Check. Laughable ending? Check.

Now I'm not saying the game was bad - far from it. But you can't say that game is 2% off perfect with a straight face.

[Serapth]

If I recall, for like 6 years, Alpha Centauri was the highest ranked game in the history of PC Gamer with a 99% rating, or maybe even 100%.


Then again, Alpha Centauri was a brilliant game.

[Shifteh]

I agree, games can be awesome, but, and I say this as a Morganite - The game was far from perfect. Score in the 90's? Sure. 99 or 100? No.

[Kefkataran]

I know CGW, but what is GFW?

GFW = Games For Windows. It was just announced last week that Computer Gaming World would be changing its name to this some time soon. I'll be interviewing Jeff Green for this Friday's episode of the EvAv Radio show about this, so watch for it if you're interested.


If you give a ranking like 98%, there had better be nothing, and I mean nothing wrong with that game. Game on rails? Check. Mediocre story? Check. Terrible teammate AI? Check. Mediocre enemy AI? Check. Laughable ending? Check.

Non-linear gameplay isn't essential for me to enjoy a game? Check. I actually liked the subtle but interesting story where you have to actually dig to find out what's going on? Check. I thought both the teammate and enemy AI was quite good? Check. But I suppose in your world, people can't POSSIBLY disagree with you, hm?

[31 Flavas]

Metafuture sounds like a great site to keep an eye on. If they drum up enough attention, there could actually finally be that much-needed push to make 5.0 = average like it should.The problem isn't that 5.0 should be average... it's that we grade games on an academic scale.

How manytimes have you looked at a game review, seen that it scored 6.x out of 10, and dismissed it without a second thought? I know i'm guilty of that.

[DeadScreenSky]

If you give a ranking like 98%, there had better be nothing, and I mean nothing wrong with that game. Game on rails? Check. Mediocre story? Check. Terrible teammate AI? Check. Mediocre enemy AI? Check. Laughable ending? Check.
Not to mention the terrible bugs all over the place. I still get occasional audio hitching in a lot of Source games.

[Kefkataran]

Not to mention the terrible bugs all over the place. I still get occasional audio hitching in a lot of Source games.

And once again, I've never had a single problem with bugs. There's ocassional audio repeating when I first load a level, but that's about it.

[Spigot]

What if we got rid of scores altogether?

Imagine for a minute, a well thought review where instead of rating, it simply lists pro's and cons, perhaps outlining them in a seperate segment. At the end of the review the reviewer either says he liked it, didn't like it, or if the target demographic would. Then the reader is armed with what the reveiwer thinks personally, the pros and cons, and is able to make up their own mind.

Just a thought.
This is actually the thing I really like about Gamespy's reviews, a pox be upon them. If nothing else, I can usually glance at the pros and cons boxes at the top of the review and see if there is anything glaring. Granted, it is still good to read through the reviews of multiple sites to see if the same problems keep surfacing and then making the decision on that basis.

And just to show that not everyone feels the same way about games, I recently was in a rather heated arguement with my brother about RE4. He HATED it. It took me a while to get my jaw off the floor and try to wrap my head around the idea of someone actually calling that game, of all the Resident Evils, a crappy one. His reasons are far too picky and dumb to mention, and he isn't usually as completely off the mark as far as falling in line with conventional thinking about good games. I think what he disliked was the fact that it really went in a different direction from the earlier RE's whereas that was the element of the game that really appealed to me, seeing as I was, at best, a casual fan of the series prior to 4.

[Demo_Boy]

Actually there should only be a 3 point scale.
1 avoid
3 must buy
2 promising but not the best get if you like similar.

[Baron Samedi]

It works. It won't change. Americans hate to be strictly average, so just skew the review scores and BAM!!! We look above average on other scales.

[ProfPuppet]

I'm liking that site a lot. I was looking at some gaming magazine the other day and it said that 7 was 'Average'. On a scale of 1-10. I place part of the blame on the magazines/sites, and part of it on the corporate sucking they have to do to get exclusives and games sent to them.

[Mason]

I don't think that the review sites are the problem.

Look at Dungeon Siege II's expansion. It's getting a pretty standard 60% for being short and not adding a ton, but still being a competent hack-and-slash. Now, by the logic espoused in this thread (academic grading only allows scores between 70% and 100%), the DS2 expansion would necessarily be one of the worst atrocities ever put down on plastic.

Is it? Can we really not envision a worse game? I can personally imagine far, far worse. To say that a fair ranking system would put the short and flawed yet basically competent DS2 expansion somewhere in the 8% range is awfully presumptuous.

Nor should 95+% be reserved for a single, radiant game that represents the sum of all human endeavors. To claim that a game is great is not to also claim that it is impossible to find fault with it. Nothing is ever beyond potential nit-picking, but a perfect score is reasonable if the flaws are trivial in comparison to the quality of the experience. It's a bold claim, certainly, but not an unthinkable one.

[DeadScreenSky]

And once again, I've never had a single problem with bugs. There's ocassional audio repeating when I first load a level, but that's about it.
So how does your almost bug-free experience erase the situations of many, many other gamers? Audio hitching with HL2's initial release was terrible for a huge number of gamers, and Valve created numerous patches to try and get it under control. It wasn't some kind of rare problem - you can even hear it in some of Valve's E3 announcement footage!

[Spigot]

I don't think that the review sites are the problem.

Look at Dungeon Siege II's expansion. It's getting a pretty standard 60% for being short and not adding a ton, but still being a competent hack-and-slash. Now, by the logic espoused in this thread (academic grading only allows scores between 70% and 100%), the DS2 expansion would necessarily be one of the worst atrocities ever put down on plastic.

Is it? Can we really not envision a worse game? I can personally imagine far, far worse. To say that a fair ranking system would put the short and flawed yet basically competent DS2 expansion somewhere in the 8% range is awfully presumptuous.
Hey! I just picked up the DS2 expansion. Stop the hate! :)

You've basically given voice to my issues with game scores of late. 60% is concidered bottom of the barrel, while I can recall game scores in the late 90's in some magazines and sites hitting single digit percentages for the real turds. 60% for an expansion that basically adds a few new character classes, spells, items and an extra act while doing nothing new is actually quite fair to me. If it had said that it broke the game, messed up the balance and could be finished in less than 3 hours, that would be a gamebreaker to me.

And I agree with your statement about how perfect scores don't mean that they're the best game EVER. They show that the game is not only fun, but usually technologically advanced or innovative and well executed. A game that may have warranted a perfect score one year will be seen as quaint and underwhelming the next depending on the advances in genres, tech and general gameplay innovations. However, games that do hit those perfect scores should be games that resonate in the industry for a time after they are released.

[31 Flavas]

Actually there should only be a 3 point scale.
1 avoid
3 must buy
2 promising but not the best get if you like similar.Well, my idea is along those lines. It'd be either a 3 or 4 rating scale, something like this.

- Lots of fun
- Fun
- Not my thing / could be better
- It sucks!

It eliminates the tendancy for the reader to judge the game based an 90-80-70-60 scale and simplifies the whole judging process. What better way to judge a game then on ultimatly, all things considered, how fun it is? Was it lots of fun? Fun? Could have been better or does it just suck?

Then in your review should the read actually want to soil themself with it, you can detail what parts you thought were better or worse or why touch screen is gimmicky or whatever.

[Spigot]

I remember an old magazine or site that would rank games based on how much people would be willing to spend on them.

A "Buy at full cost" recommendation would mean that the game was good enough to buy sight unseen on launch day.

"Wait for the bargain bin" would be for the games that were good, but probably not worth full price.

"Rent before you buy" would be for those that needed to be played first before a decision to fork over the actual retail cost could be determined.

"Not worth a rental" pretty much sums it up. For the real stinkers.

There was also one that used a dollar value to signify how much the game would be 'worth' as a rating, but I forget exactly how it was broken down.

[DeadScreenSky]

The four point scale a lot of people are suggesting is the same kind of thing Daily Radar used. Their system went (I think) Direct Hit, Hit, Miss, Dud. It was a great system.

[Kefkataran]

So how does your almost bug-free experience erase the situations of many, many other gamers? Audio hitching with HL2's initial release was terrible for a huge number of gamers, and Valve created numerous patches to try and get it under control. It wasn't some kind of rare problem - you can even hear it in some of Valve's E3 announcement footage!

Because a review I'm writing of the game would pretty necessarily be based off of my opinion which would be pretty necessarily based off my experience with the game?

[Spigot]

The four point scale a lot of people are suggesting is the same kind of thing Daily Radar used. Their system went (I think) Direct Hit, Hit, Miss, Dud. It was a great system.
That's where it came from! I loved that system. Very succinct and when followed up in the review, it was easy to see whether the game was worth picking up or not.

What drives me nuts is when people have a review that is generally negative and then post a high score, or vice versa. If I can't follow from the write-up why you came to the score you gave a game, the score should be thrown out and the review rewritten.

[MosBen]

I'll throw my hat into the ring for the 4 point scale, though really any system can fit that if we understand how it's being used. All I need to know from the points is the general quality of the game and the four point system does that rather well. Still, with the ten point scales that most sites use now I get pretty much the same thing: 9.0-10.0 is a fantastic game, 8.0-9.0 is a good game or a great game with some flaws holding it back, 7.0-8.0 is a mediocre game or a good game with some flaws, and below that are games that are either pretty boring or so flawed that they're unplayable. As much bitching as people do about that system, I think we all know exactly what one of those scores means every time we see it, and really, that's the point of the point system; to give you a really quick reference to the game's quality if you don't have time to read the review which will explain it.

Ultimately, if you understand the system and it is consistently applied then it serves its purpose. If it's not consistently applied, well, then the quibble is in the application, not the system.

[DeadScreenSky]

Because a review I'm writing of the game would pretty necessarily be based off of my opinion which would be pretty necessarily based off my experience with the game?
Come on - adequate PC reviewers run the game on a whole bunch of machines all using different hardware. There's no way it was missed.

[Kefkataran]

Come on - adequate PC reviewers run the game on a whole bunch of machines all using different hardware. There's no way it was missed.

I suppose big magazine reviewers can afford to do that. Either way, I've just never had problems with it, so I still fail to see it as a major "MINUS POINTS" for Half-Life 2, and we were talking about me giving it a score in the high 90s.