Metacritic = Meaningless

[Baron Samedi]

In an editorial (http://www.gamasutra.com/php-bin/news_index.php?story=18562) over at Gamastura, Keith Boesky assays Metacritic and determines that it is "indicative of neither quality nor sales":

Orders are influenced by a buyer’s review of the game, the publisher’s marketing commitment to the game and the publisher’s pipeline of future titles. The hundred million dollars in movie marketing money will drive more orders than than a 90 for Psychonauts. If the big marketing budget is coupled with a product from the company about to release Call of Duty 4 - even better.

Large marketing, strong publisher pipeline, means large order. If those buyer commitments are big, projected revenue increases and with it, projected marketing, sometimes leading to incremental orders, most of the time leading to stronger consumer awareness and therefore stronger sales. All of these events occur while Metacritic still has an N/A next to the game.

GTA4 = 98 = Meaninglessness?

Editor's Note - I changed the title from "Borked" to "Meaningless" to avoid confusion. The Metacritic site is not broken or down

[violentp]

I always thought it was. Especially when people try to make points based on it.

[Telefrog]

Well, it's certainly meaningless in terms of sales, but it's a pretty good source for quick quality judgements.

I go to MetaCritic/Gamerankings to look at the aggregate scores of a game and then, if I'm still interested, I'll go to the individual reviews to see why a game scored a particular way.

[Baron Samedi]

I highly suggest you read the entire article before you post in this thread. It's a long read, but worth it.

[Yellowman]

Ok, I didn't read the article but I think that it's interesting to note that GTAIV is just a hairs breadth above Zelda on Gamerankings. Like Galaxy it has suffered from the 'too many reviews' syndrome.

[Exodus]

With games like GTA4 though if you were a fan ever since 3, you're more than definitely gonna get 4.. and anyone who hasn't liked gta since 3 well, gta4 is just the same stuff but damn does it look nice

[Telefrog]

I highly suggest you read the entire article before you post in this thread. It's a long read, but worth it.

I did, and this is where the argument fell apart:


So, back to the Jerry Maguire moment. I guess we really shouldn’t rely on the Metacritic as an indicator of quality or shareholder value. Quality should be measured the old fashioned way: sales.

50 Cent: Bulletproof says thank you.

[Norse]

So if a good Metacritic score doesn't indicate quality, what does? Single review scores doesn't say much, but an overall score is a pretty good indicator IMHO

[roboninja]

I love metacritic and gamerankings for quick indications of the quality of a title. I do not take the scores too seriously, in that I would not buy one title over the other simply because one had 10 more points on metacritic. But when I see a game that is at 40, that is usually a good indication the game sucks. They are good tools, if you use them right.

[Wolvie]

I don't take these types of sites at face value. Take Gamerankings.com for example. If I were to believe it, Zelda:OoT was the best game made of the last ten years. And I personally don't feel that it is. All these sites take into consideration is a number.
But a game that was great back then might not be as good today. If one was to truly judge a game, they would look at reviews, sales, fan opinion, perhaps even compare it to other games of the genre to see how it stacks up. In other words, it's a tough thing to do when judging games. Personal opinion is always best in this case.

I love metacritic and gamerankings for quick indications of the quality of a title. I do not take the scores too seriously, in that I would not buy one title over the other simply because one had 10 more points on metacritic. But when I see a game that is at 40, that is usually a good indication the game sucks. They are good tools, if you use them right.

Absolutely. If a game has a ratio of 30% then that must be one shitty game. This is the only reason I look at these sites, to see how good/bad others think the game is.

[inscribed]

didn't read the article, but i always make sure i check metacritic before making any purchase. i might not base my purchase solely on whatever total score they give a game, but the list of reviews they provide for each game is invaluable.

[Serapth]

As the only metric, yes, it obviously is. That said, its one statistic that is perfectly valuable and only an idiot would ignore.

[violentp]

As the only metric, yes, it obviously is. That said, its one statistic that is perfectly valuable and only an idiot would ignore.

I beg to differ because even if a game sold 50 million copies and is ranked #1 on Metacritic, it still says absolutely nothing about whether or not I'd enjoy it. And since I don't own any stock in any game companies, the sales numbers mean nothing to me therefore, I'm an idiot.

[vherub]

I read the article, and I don't understand this "publisher has a strong pipeline" statement.
Capcom is a pretty big game publisher, they should have a strong pipeline, and yet they released a really good game (Okami) on two of the most successful consoles ever, the ps2 and wii. That game seems to have sold less than inferior games from smaller publishers with less marketing.

[Serapth]

I beg to differ because even if a game sold 50 million copies and is ranked #1 on Metacritic, it still says absolutely nothing about whether or not I'd enjoy it. And since I don't own any stock in any game companies, the sales numbers mean nothing to me therefore, I'm an idiot.

Yes, as I said it cant be the only metric.

That said, you as a consumer know that if a game gets a high ranking on Metacritic, most people like it. That means it is most likely fairly bug free, etc...

[Johan]

Tell that to MS, which is going to delist Arcade titles based in part on Metacritic scores.

:(

[Baron Samedi]

50 Cent: Bulletproof says thank you.

That's out of context. 50 Cents is an example of the disconnect between Metacritic and sales. I could raise BG&E or SotC. From the same quote:

So, back to the Jerry Maguire moment. I guess we really shouldn’t rely on the Metacritic as an indicator of quality or shareholder value. Quality should be measured the old fashioned way: sales.

Quality as he's using it != the quality of being good, but the quality of moving copies.

[Serapth]

Tell that to MS, which is going to delist Arcade titles based in part on Metacritic scores.

:(

... that said, it weeds out the shit, which unfortunatly there is too much of on live Arcade.


However, there is an area Metacritic falls on its face. Retro games should not be rated as reviewers are pretty much shitty at it. I mean, by todays standard how do you rate Ms PacMan or Tapper? Really it boils down to do you or dont you like the game...

[Telefrog]

Yes, as I said it cant be the only metric.

That said, you as a consumer know that if a game gets a high ranking on Metacritic, most people like it. That means it is most likely fairly bug free, etc...

Agreed. I know that I don't like every high-scoring title on Metacritic, but if I see that a game has a really high or really low aggregate score, that's a pretty good indicator to me of its overall quality. At the very least, if I know nothing about a game, it lets me know that buying one with a high score is a safer bet than one with a score in the 40's or below.

[Telefrog]

Quality as he's using it != the quality of being good, but the quality of moving copies.

That's great if you're a shareholder, but as a consumer it's a pretty shitty way of judging quality. Yes, I know the article is specifically talking about using aggregate review scores within a publishing company to determine payouts and milestones, which I have no argument with. If your job is to make money, and people buy 50 Cent games, then by all means make a hundred 50 Cent games.

My argument is that sites like Metacritic and Gamerankings were not originally made for game/movie/book publishers to internally rate their products. That some do, is not my problem.

As a consumer, my job is to maximize my gaming money. I don't care about their sales, I care that I'm getting a quality product that I like. Metacritic helps me to do that.

[Micasa]

A good (or bad) Metacritic score might not influence first-day sales, but I think it's naive to think it doesn't affect long-term sales.

[Baron Samedi]

My argument is that sites like Metacritic and Gamerankings were not originally made for game/movie/book publishers to internally rate their products. That some do, is not my problem.

threadjack/

Okay, so some publishers skew Metacritic scores to gain sales...

As a consumer, my job is to maximize my gaming money. I don't care about their sales, I care that I'm getting a quality product that I like. Metacritic helps me to do that.

So Metacritic helps you, despite having skewed scores?

What's interesting is that GTA4's critic score, based off of 69 reviews = 98; whereas the users score, based off of 1013 reviews = 82. Disconnect?

[the soUL TRAder]

It's a good resource to know if a game you were interested in because of it's gameplay and story, was properly executed or not.

[Telefrog]

threadjack/

Okay, so some publishers skew Metacritic scores to gain sales...?

So Metacritic helps you, despite having skewed scores?

What's interesting is that GTA4's critic score, based off of 69 reviews = 98; whereas the users score, based off of 1013 reviews = 82. Disconnect?

Which titles would be the ones with scores skewed by the publisher? I'm willing to bet that most, if not all, instances of gaming in the scores is done with titles that were "on the bubble" anyway. What amount of gaming would've influenced the aggregate scores of GTA? Zelda? Haze? Kane & Lynch?

That's why aggregate scores work better than a single review. No publisher has the time, money, or influence to hand out moneyhats to everyone.

[Zanch Effect]

I've always seen it as pretty indicative of quality. It just puts all reviews into a combined score. How is that broken exactly?

[oldjadedgamer]

Metacritic isn't meaningless to the publisher. After all, Sony complained to GameSpot about the 8.5 they gave Ratchet and Clank which many thought was totally fair for the game and a good over all score.

[AniAko]

Well it's meaningful now that Microsoft is using the Metacritic rating along with some other stats to determine if you'll be "shoved off" XBLA

[Kelegacy]

I always go to Gamerankings, look up a game, and read the reviews listed, the good and the bad. It's always good to have a sampling of opinions. I do this for new and old games alike. I still bought Jericho recently despite the middling reviews ($15). I find myself caring less and less about reviews as I grow older, whereas in the past I followed them religiously.

Yet, if a game I'm interested in gets mediocre or poor reviews, I won't buy it right away. I may still grab it, but only after a price drop or two. So they are still important to me for the AAA day-one buy type of games. At $60 a pop, I can't afford to make too many mistakes.

[MJBuddy]

This is a flawed judgement - yes a strong marketing plan is key, but the finished to-the-shelf/theater product is as well.

Look at movies like Speed Racer, which had so much marketing you could have a seizure while watching television. The movie has yet to recoup half of its cost, and its marketing structure had 80 million dollars backing it.

Had the day 1 reviews been "omg amazing - you absolutely must experience this film", it would be crushing its budget. But they weren't, and it isn't.

[Kelegacy]

This is a flawed judgement - yes a strong marketing plan is key, but the finished to-the-shelf/theater product is as well.

Look at movies like Speed Racer, which had so much marketing you could have a seizure while watching television. The movie has yet to recoup half of its cost, and its marketing structure had 80 million dollars backing it.

Had the day 1 reviews been "omg amazing - you absolutely must experience this film", it would be crushing its budget. But they weren't, and it isn't.

How do you explain Alvin and the Chipmunks and all the other terrible movies that still make hundreds of millions, despite having poor reviews?

Bad games, books, movies, music...they still sell like hotcakes if you can snag hold of the main stream. Actually, I think success for quality entertainment has to do with a great deal of luck.

[Isamura]

There are a lot of stupid shoppers out there who get dupped by marketing. Luckily, we, the informed minority, have metacritic to let us know when a game is crap.

[Heretic Machine]

Meh, this guy is an idiot... it's as simple as that. First of all, while review scores aren't perfect, they aren't even approaching being worthless when judging whether or not a game is good (and by good, I mean that people who will enjoy the genre will enjoy the game, more than likely). Second, while sales might not be entirely dictated by scores, I'm positive that they must have some effect, since I often avoid games based on review scores. I seriously doubt that I'm the only one in the world who checks out reviews before buying a game.

It sounds like this douche just wants to shut out the gaming media (irony), and let marketing dictate the future of gaming. We already did that, it didn't end well (http://en.wikipedia.org/wiki/North_American_video_game_crash_of_1983). Turns out, people don't like being duped into buying shitty games over and over again, and eventually stop buying them all together.

Let's apply this to another industry: The movie industry. Imagine if there were no movie critics, just advertisements. Every movie made would be on par with Spider-man 3, and it would take all of six months for DVD and ticket sales to bottom out entirely as people realize that their time and money is better spent elsewhere. Contrary to popular belief, the average American consumer isn't completely blind, and will not patronize something that they don't find any value in.

[handsalad]

Doesn't his argument fall apart when you look at a game like Okami on the PS2. It has a 93 metacritic but the studio that made the game was closed down due to crappy sales. Sales do not indicate quality nor does marketing dollars spent or hype generated.

[Jesse_Divnich]

Some data provided by EEDAR. Doesn't come to any conclusions, but sheds light on the topic.

http://www.gamasutra.com/images/divnich_0408/metacritic-by-publisher-whiskerplot-4999.gif

http://www.gamasutra.com/images/divnich_0308/MetaCriticGraph.jpg

[Talanvor]

Um, and where do we get these magic sales numbers? The ones that we have to guess at? Or the ones that only report what the retailers bought?

And in that case, we all should own every iteration of Madden. I mean, if it sold so much, it must be great, right? We buy it because it's great, it's great because we bought it. Nice little circle that guy just drew.

[Jesse_Divnich]

Um, and where do we get these magic sales numbers? The ones that we have to guess at? Or the ones that only report what the retailers bought?

And in that case, we all should own every iteration of Madden. I mean, if it sold so much, it must be great, right? We buy it because it's great, it's great because we bought it. Nice little circle that guy just drew.

Unfortunately, I can't post exact sales figures, but the graphs are based off of NPD data. But if you want to convert the 0% to $1 and work from there.

A wii game with a 90%+ quality score sells $7 for every $1 of an average game sells, but I don't think that part is all that important to the subject.

Interesting part is that Wii games in the 50-59 range sell more than Wii games in the 60 to 80 range. Personally, I think it has a lot to do with marketing and branding when it comes to the Wii and its casual market base.

[Neosho]

I don't take these types of sites at face value. Take Gamerankings.com for example. If I were to believe it, Zelda:OoT was the best game made of the last ten years. And I personally don't feel that it is. All these sites take into consideration is a number.
But a game that was great back then might not be as good today. If one was to truly judge a game, they would look at reviews, sales, fan opinion, perhaps even compare it to other games of the genre to see how it stacks up. In other words, it's a tough thing to do when judging games. Personal opinion is always best in this case.

I think that you're totally missing the point of how you should be using metacritic type sites. If you don't like zelda style games, you're not going to look at OoT and say "wow, this is a great game".

Metacritic provides a value, it's up to you to contextualize that value as it relates to your desired gaming experience...and personal opinion, IMO, is really the wrong way to go about that. Plenty of members on this site wax poetic about STALKER, but as much as i wanted to love the game, it was a buggy piece of shit that only delievered 1/10th of what it was supposed to in shit-tastic fashion. A site like metacritic will minimize the fanboyism in almost all cases, so if you look at a game and say "i wonder how this represents the genre", a metacritic number is probably the most meaningful number you can have.

It's still nothing against playing a demo or the like, but it's the best of the worst, sorta ;)

[MJBuddy]

How do you explain Alvin and the Chipmunks and all the other terrible movies that still make hundreds of millions, despite having poor reviews?

Bad games, books, movies, music...they still sell like hotcakes if you can snag hold of the main stream. Actually, I think success for quality entertainment has to do with a great deal of luck.

Alvin specifically? Kids still loved it. Parents don't read reviews before they drag their children. I think one of the problems Speed Racer specifically had was that it didn't convey that it was a children's movie enough.

Nothing will sell without *some* good word of mouth. I discuss games with my friends - if they all pan something I was on the fence about, I'm probably not going to waste my time. I can't speak for books, I really don't pay attention to the market there much, but everything else ends up being a matter of taste, or strong marketing with a weak payoff hurting monster sales. The best example of marketing *selling* the movie is Blair Witch. The best example of the movie selling the marketing? Right now Iron Man, which a bunch of people on this very website panned before it even came out. The reviews poured in favorable and they caved.


I'm not saying marketing can't sell, I'm just saying the ration of marketing dollars/positive reviews is pretty high. The better the movie, the less marketing you need.

[dr_wily]

is it me or is metacritic almost always lower than gamerankings..

im a GR guy myself.

[Hemalin]

is it me or is metacritic almost always lower than gamerankings..

im a GR guy myself.

The reviews that each site decides to count toward the overall score differs slightly between the sites. Depending on which sites they include/exclude causes the difference.

[Variable Gear]

Metacritic is meaningless because reviews are worthless.

[Apushmataha]

Interesting part is that Wii games in the 50-59 range sell more than Wii games in the 60 to 80 range. Personally, I think it has a lot to do with marketing and branding when it comes to the Wii and its casual market base.

That would be my guess. The general consensus with my non-gaming friends is that any game that isn't "casual" is too hard for them, even without giving it a try. The games that are typically critically rated well are somehow considered to be more "hardcore". I mean, really, is a game like Elebits (rated a 75 on metacritic) really more hardcore than Carnival games?

I wonder where that logic came from.:confused: