Developers Crying Over Next Gen

[Ridlin]

DailyGame.net has an interesting article (http://www.dailygame.net/news/archives/004423.php) up that trys to explain (in plain english) why things like in order CPUs and multicore programming is so difficult for developers to master.


You've probably heard a lot of grumbling about the next generation of consoles having In Order CPUs - allow me to try and explain why. Today's modern PCs have what's called an instruction window (extra memory that's built right into the CPU - cache). This instruction window is a holding place where code can be optimized and rearranged before hitting the CPU.

Let's use a doctor's waiting room as an example. The whole building is the CPU, but the doctor is what actually treats the code. Imagine the office secretary coming out into the waiting room and organizing the code waiting to see the doctor. The secretary organizes them in an optimal way for maximum speed, determines who gets in first and who doesn't need to get in at all. As this does in a real doctors office, it speeds up things a lot. An entire family of code will enter the waiting room and the secretary will throw half of them out.

[Borys]

Just like PC devs can't really grasp the idea of coding games for multiCPU/ multithreaded systems (Uni's papers and labs won't be enough) the console devs (real console devs not PC/ Xbox ones) can't grasp the idea of pixel and vertex shaders even if they would bite them in the ass.

So it's all in balance.

[EvilBob46]

One thing I'm wondering: How does a reasonable priced single-core processor, like a mid/lower end Athlon 64, compare to the processors being used in the XB360 and PS3?

Somehow, I doubt the performance difference is going to be all that huge (EDIT: although the other processors would probably get a lot more performance after vigorous optimization), and the single core processors are going to be much easier to develope for and potentially cheaper if you choose a model with a decent price/performance ratio.

At least it could be something, say, Nintendo should consider for its console?

[Opty]

I've said from the start (well, from when the other two were revealed to have in-order CPUs) that if Nintendo goes with an out-of-order processor they're going to make the developers so much more happy that they'll get a large chunk of 3rd party support just because of that.

[Zeal]

This must explain why most next-generation titles look and run like crap.

i.e., Perfect Dark Zero.

[Hellstorm]

The Revolution is coming to save the day.

[Ridlin]

Somehow, I doubt the performance difference is going to be all that huge

They're discovering that code they've written that words fine on a PC doesn't work at all on an in order CPU, or that it's so slow it seems to have locked up. That's what Gabe N is so mad about.

It's the multiple processor coding that's really kill'em, though.

[bait]

Hard to program consoles are bad for the entire industry.
It sounds like developers aren't communicating very well with machine makers (or they are unwisely being ignored). I want consoles to be as easy to program as possible for many reasons:
1. Core mechanics of the game can be completed easier so more time can be taken for optimization, bug chasing, and just making a better game.
2. It's easier for new and fresh thinking companies (without truckloads of money) to get in the game. Programmers who can master this silly scheme will be fewer than previous generations which means developers will have to pay them more and that means higher game prices.
3. With higher development costs, it's also more risky to try daring new ideas. This means more sequels and "me too" titles. Creativity is strangled.

Whichever one of the big boys wants to listen to the developers and make their console the easy one to program for will reap the rewards.

[51|RandoM]

The real question is whether it is inherently difficult, or is it just a matter of developing new tools, new methods, and manageing the related learning curve.

[Zeal]

This is also a very well written article, I might add. The guy really knows what he's talking about and his experience in the industry shows.

I also like how he killed the hype of next-gen consoles being able to produce Toy Story graphics. His views are a refreshingly realistic take on where we are in terms of graphics and hardware. http://www.evilavatar.com/forums/images/icons/icon14.gif

[president_fred]

Very good article and thought provoking I hope the big N decide to include an instruction window, talkin'bout a revalution, yeah! Or not. I hope this won't mean that the first 2 years suck balls, then I may just wait I was looking forward to getting at least one console at launch.

[Mr.Green]

Hi, my names Robert Dusseau, and I've been programming steadily in C++ and Java for over 11 years now. I had started off trying to make games way back in the day, and I did managed to create a little 2d fighter that made me proud, but then Doom came out, and after that trying to catch up with the ever changing field of game development proved to be too painful for someone with a family.

So I accepted a less stressful job developing inventory and communications systems for a large company in Michigan. After 10 years of doing that, I now lead a team of other programmers and have what I feel is a lot of insight into what's going on in the gaming world, especially since I study and write about it most everyday.
The guy really knows what he's talking about and his experience in the industry shows.
Experience in the industry? Yeah sure, except not in the gaming industry. I have a similar profesional background, I've been working on mods since Doom, but I won't pretend I'm an expert in game developement. This guy doesn't seem to have a problem with that though.

Anyway, of course it's going to be harder and you're going to hear a lot of cries from the technically challenged. People are affraid of change, that's the way it is. Just give it time.

[danhoo]

I've said from the start (well, from when the other two were revealed to have in-order CPUs) that if Nintendo goes with an out-of-order processor they're going to make the developers so much more happy that they'll get a large chunk of 3rd party support just because of that.

and

Whichever one of the big boys wants to listen to the developers and make their console the easy one to program for will reap the rewards.

Sadly, I wish this was true, but console developers do not drive what gets developed currently -- publishers do. And publishers are mostly concerned with what console has the biggest install base, not what console is easiest to develop for.

At least for the 'last' generation, all the publishers I ever worked with insisted on a PS2 version first, even when we explained that an XBox version would be easier to do (and therefore quicker to market), because the PS2 had a nearly 10-to-1 install advantage.

If a console maker made their hardware easy to program, _and_ somehow opened things up so homebrew stuff could be easily distributed, then they might start to see more developer-driven stuff. But this is asking a lot.

[chrisg]

Experience in the industry? Yeah sure, except not in the gaming industry. I have a similar profesional background, I've been working on mods since Doom, but I won't pretend I'm an expert in game developement. This guy doesn't seem to have a problem with that though.

Anyway, of course it's going to be harder and you're going to hear a lot of cries from the technically challenged. People are affraid of change, that's the way it is. Just give it time.

Sorry, but code is code and if the output is a game or an application does not matter here. With his background he prolly knows his tools, the language, what it can and what it cannot do, how the compiler translates the code, etc...
The developers trying to hide behind the argument that game programming is fundamentally different from everything else are just trying to steer attention away from the fact that multithreading is done in all the other parts of the IT industry for years now and that they finally have some catching up to do.

They act as if SMP never existed, yet people in other parts of the industry are developing for exact multiprocessor systems for years. They should stop complaining and get back to work. Multithreading is hard no doubt and its sometimes all but fun, but in the end its nothing that you cannot master in a reasonable timeframe. There will be A LOT of hurdles to overcome, but that is what the fun in programming is all about and what drives a lot of developers in the IT industry...

[dolemite]

The real question is whether it is inherently difficult, or is it just a matter of developing new tools, new methods, and manageing the related learning curve.

Writing code for multiple processors is inherently difficult, and judging by the new hires we've gotten, it's not something that's covered very much, if at all, at the undergraduate level.

[Rangoth]

I have a feeling that, like it or not, multi-core CPU's are not going away any time soon.

[Metal]

Sounds to me like EPIC was the only PC Focused Development house smart of enough to listen to the hardware manufacturers and actually develop an engine for multicore processors. Valve and ID are getting waxed by Epic right now. The Doom 3 engine looks dated already in comparison and Source never did impress me technically. I think they did a lot of things in Half Life 2 to make up for the fact that the engine is a little weaker than what Doom 3 was. They seemed to have a much better art team than ID did.

[CarpeAmentum]

Writing code for multiple processors is inherently difficult, and judging by the new hires we've gotten, it's not something that's covered very much, if at all, at the undergraduate level.

I guess I can't speak for other degree programs, but I am going into my Senior undergraduate year in a Computer Science program, and we have covered parralell design, though limited (1/4 of a class) and I think we will get it some more in one or two more courses... not alot ... but then again, what is covered alot? There are, after all alot of things to cover...

[Stryfe01]

Hard to program consoles are bad for the entire industry.
It sounds like developers aren't communicating very well with machine makers (or they are unwisely being ignored). I want consoles to be as easy to program as possible for many reasons:
1. Core mechanics of the game can be completed easier so more time can be taken for optimization, bug chasing, and just making a better game.
2. It's easier for new and fresh thinking companies (without truckloads of money) to get in the game. Programmers who can master this silly scheme will be fewer than previous generations which means developers will have to pay them more and that means higher game prices.
3. With higher development costs, it's also more risky to try daring new ideas. This means more sequels and "me too" titles. Creativity is strangled.

Whichever one of the big boys wants to listen to the developers and make their console the easy one to program for will reap the rewards.


It's all BS. I'm not disputing that it's hard. Still so was the PS2 a pain in the ass to develop for with shitty support from Sony to boot, but the PS2 still came along and got plenty of support . Just like with all things, after they have some time with it it will get easier. Even PC's are going to be all multicore soon, so they will ahveto learn or be out of work.

[Ridlin]

Right, all the article was saying was give them some time. They'll get it. Don't expect to be blown away immeditatly.

People who want to understand why it's such a big change can understand why.

[bait]

and



Sadly, I wish this was true, but console developers do not drive what gets developed currently -- publishers do. And publishers are mostly concerned with what console has the biggest install base, not what console is easiest to develop for.

At least for the 'last' generation, all the publishers I ever worked with insisted on a PS2 version first, even when we explained that an XBox version would be easier to do (and therefore quicker to market), because the PS2 had a nearly 10-to-1 install advantage.

If a console maker made their hardware easy to program, _and_ somehow opened things up so homebrew stuff could be easily distributed, then they might start to see more developer-driven stuff. But this is asking a lot.

I agree, this is sad. That's why the #1 console of this generation doesn't have the best graphics, doesn't have the best sound, doesn't have the best support, doesn't have the best engineering, doesn't have the best reliability, and is a pain in the ass to program. I know the publishers go to the PS2 for the fat installed base, but is this really a good trend for the industry?

[51|RandoM]

There is always the option of sticking with one core in multicore, or one cpu in multicpu. Ah, but then the company that actually does the work to get up to speed on either platform is going to steal your lunch.

I just see somebody whining about having to maintain a competitive advantage.

It seems like it should be even easier than the old consoles that had dedicated chips for different tasks. If they could program to that division of tasks, why not do it again, just apply the division logically.

[Mason]

There is always the option of sticking with one core in multicore, or one cpu in multicpu. Ah, but then the company that actually does the work to get up to speed on either platform is going to steal your lunch.

I just see somebody whining about having to maintain a competitive advantage.

It seems like it should be even easier than the old consoles that had dedicated chips for different tasks. If they could program to that division of tasks, why not do it again, just apply the division logically.

If a company needs to double its number of programmers to make a competitive game, that increases the cost of making the game. There isn't a huge amount of new people getting into gaming the way there was in the past few decades, so you can't anticipate game sales to go up a lot above that of the current generation.

So, if it costs more money to make the game but it doesn't end up making more money in sales, making games isn't a very good business. It isn't a good business to work in (so talent will flee to more lucrative industries) and it isn't a good business to invest in. The megastudios will survive, obviously, and everyone will get their next GTA, Halo, FF, and EA sports games. But the industry itself will be a zombie, too stupid to know that it is already dead.

And coming up with algorithms for parallel processing is not a trivial task. The standard pipeline for games doesn't lend itself very well for parallelism. The hardware choices for the XBox 360 and the PS3 only make sense if all you really want is a prettier version of today's game. And the only surviving parts of the industry are perfectly happy with that, since trivial remakes are what they love best.

[Mason]

The real question is whether it is inherently difficult, or is it just a matter of developing new tools, new methods, and manageing the related learning curve.

The exact same thing could be said of the Native Americans when the Spanish showed up. "Hey of course you can compete and survive, just start settling in major population centers fueled by a stable agrarian society, build up an educated class that can engineer the technologies needed to produce an industrial base, and whoops wait they slaughtered you and took your land."

For small and middle sized developers who already stretch themselves to the limits to produce a game that rarely sells more than it costs to make, putting challenging new curves up in front of them is basically homicidal.

[RMan]

Sounds to me like EPIC was the only PC Focused Development house smart of enough to listen to the hardware manufacturers and actually develop an engine for multicore processors.
There's no real indication that their stuff takes advantage of the hardware any better than the other guys, they're just politically minded enough not to step on toes like id and Valve seem to do (I’m sure you’ll hear plenty of bitching about hardware and drivers if you were at Epic’s office, it just doesn’t generally leave there). It is strategically bad for any middleware developer to bitch about multi-core development (or any hardware, for that matter), it implies a lack of interest in making your software work with it and there’s absolutely no upside. In the end you’ll mostly just sound whiny and short sighted, no matter how right you may be. You can hate and fear it all day long, but you’re stuff’s going to be running on a multi-core system eventually, so it’s best just to suck it up and move on.

[motor]

The multi-core stuff is fine. Smart devs (which I like to include myself as one) can take advantage of them, a good engine can offload simulation tasks, AI, sound, etc, especially when the count is low (give me 256 500 mhz processors and things are more difficult, but a half dozen to a dozen...a game can be split that way with good planning). As for the in-order, that was a real mistake, it saved them fifty cents a chip, but forced a lot of mindless assembly optimizing on us that is a waste of our time. That, the small cache size and the slow dvd load times (and that debatable) are what suck about this generation, not the multi-proc issues.

[Oblivion]

As for the in-order, that was a real mistake, it saved them fifty cents a chip, but forced a lot of mindless assembly optimizing on us that is a waste of our time.

wow, I didn't know that? 50 cents for that? that is really dumb and I cannot see the reason behind that.

[Ridlin]

The multi-core stuff is fine. Smart devs (which I like to include myself as one) can take advantage of them, a good engine can offload simulation tasks, AI, sound, etc, especially when the count is low (give me 256 500 mhz processors and things are more difficult, but a half dozen to a dozen...a game can be split that way with good planning). As for the in-order, that was a real mistake, it saved them fifty cents a chip, but forced a lot of mindless assembly optimizing on us that is a waste of our time. That, the small cache size and the slow dvd load times (and that debatable) are what suck about this generation, not the multi-proc issues.



From the GI interview with Todd Hollenshead -




GI: Based on John Carmack’s speech yesterday, he seemed not very hot on the whole multi-processor situation.

TH: No, no he’s not. He gave a keynote at GDC two years ago where he basically said that his philosophy was, "Make the most powerful processor and the most powerful GPU." And that would give him the best ability to make the game look great.


The speech with Gabe Newell -


It goes a lot worse in a multicore world, where there's a whole bunch of stuff going on in these separate [cores], can suddenly have an impact on the entire system.

"If writing in-order code [in terms of difficulty] is a one and writing out-of-order code is a four, then writing multicore code is a 10," cautions Newell. "That's going to have consequences for a lot of people in our industry. People who were marginally productive before, will now be people that you can't afford to have write engine or game code. They can't get a big enough picture of what's going on in the box so they'll be a net negative on the project."


Gabe Newell, John Carmack. Your right, those guys don't know what the hell their talking about.

[Dracula-X]

wow, I didn't know that? 50 cents for that? that is really dumb and I cannot see the reason behind that.
Multiply by 30 million and you'll see why.

[motor]

From the GI interview with Todd Hollenshead -

...

The speech with Gabe Newell -

...

Gabe Newell, John Carmack. Your right, those guys don't know what the hell their talking about.

First of all, John Carmack wasn't talking about what was possible or not, he was talking about what gave him the most flexibility. Of course I'd rather have one 10 Ghz chips versus three 3.2 Ghz chips, who wouldn't, but you know what, one of those things doesn't yet exists.

As for Gabe Newell, I think he was going a little too far into hyperbole. Huge multi-proc is really difficult (as I said), but a half dozen threads can be used fairly effectively as long as you have a good engine written to take advantage of it, it doesn't take some kind of super genius to write and it does not by any means mean that you going to have to fire the lower third of your dev pool becuase their brains can't handle it. What he is really nervous about is processor memory issues (one processor blocking another because of wanting to use the same memory) this is the hardest part of writing a multi-proc engine and it becomes an increasing problem as processor count goes up, but at this stage of a few processors it really isn't the end of the world.

One other thing, my real grip isn't about the multi-proc stuff is hard issue (sure it is hard, I contend not a killer, but it is hard), what I and a lot of other devs are mad about is the out-of-order stuff (and the small cache), that was just the console makers being greedy. Multi-proc is the future, we are starting to run into physical limits of how much faster an individual chip can go, so the only way to increase overall speed is to increase the chip count, no one can argue with that, it is the way the world works. You can cry about it, but I'd rather roll up my sleeves and deal. Also, multi-proc is an INTERESTING problem, I don't mind working on interesting problems. What I do mind is spending 10% of my time looking at assembly code and finding out that if I swap the order of two instructions I can speed up the run-time by 0.01% (repeat a thousand times).

[RMan]

Go motor, I agree fully.

[Rangoth]

I think a problem with, not all, but many coding situations today is that the appriciation for extremly tight & efficient code is not there like it used to be. I can recall years ago when one was admired for that skill, its really just an art to be able to write really tight code. Now in many shops its just get the shit out, on time, and stable. We can streamline later. Thats just sloppy man.

[RMan]

While hacking stuff together is definitely bad, I do think the focus on extremely efficient code is lessened (at least in terms of instruction based optimization), and is an appropriate response to proper optimizations with modern CPUs. IOW, a good design for the system and proper data organization will now yield a greater performance than simply tightening up the code optimizations (cache loads are generally way more expensive and fixable than poor instruction efficiency). This is especially true when dealing with GPUs or other distributed systems, since proper batching and parallel processing can make a much greater impact than all other CPU based improvements. In the end, I think the focus of good optimization has changed fairly dramatically in the last decade.

[Rangoth]

As things continue to increase in complexity there are always more bottlenecks. I understand that efficient code may not have the biggest 'cost' in terms of what is being lost however it is my belief that it is a potential situation where gains in all areas could be made. Do I think any steps will ever be made to encourage more efficient code? Not really. Its an art form from a different era, one which is no longer appriciated.

[motor]

Devs now spend a lot of time on algorithmic optimizations, which is a more creative and interesting problem. I would much rather try to come up with a way of only looking at a fraction of the objects in the world for collision (for example using an oct-tree) then spending time trying to make my bounding box collision detector assembly language tuned for each individual platform. Usually it's a bigger win...and it keeps us from jumping out of windows :)

[Eon]

Sorry, but code is code and if the output is a game or an application does not matter here. With his background he prolly knows his tools, the language, what it can and what it cannot do, how the compiler translates the code, etc...
The developers trying to hide behind the argument that game programming is fundamentally different from everything else are just trying to steer attention away from the fact that multithreading is done in all the other parts of the IT industry for years now and that they finally have some catching up to do.

They act as if SMP never existed, yet people in other parts of the industry are developing for exact multiprocessor systems for years. They should stop complaining and get back to work. Multithreading is hard no doubt and its sometimes all but fun, but in the end its nothing that you cannot master in a reasonable timeframe. There will be A LOT of hurdles to overcome, but that is what the fun in programming is all about and what drives a lot of developers in the IT industry...

Bollocks mate. By definition almost all game coding is bleeding fucking edge development under very difficult circumstances. If you think that is the same as curling out databasing solutions for a series of retail chains then you are sadly deluded.

The focus for game code is different than for regular code. Feature driven, performance driven, stability driven code is very hard to write to schedule and managing the correct balance means that game coders are under a LOT more pressure than most of their counterparts in other industries.

[Rangoth]

Bollocks. Thats just a damn cool word. :)

[dolemite]

Bollocks mate. By definition almost all game coding is bleeding fucking edge development under very difficult circumstances. If you think that is the same as curling out databasing solutions for a series of retail chains then you are sadly deluded.

The focus for game code is different than for regular code. Feature driven, performance driven, stability driven code is very hard to write to schedule and managing the correct balance means that game coders are under a LOT more pressure than most of their counterparts in other industries.


Whoa, you might just want to tap the brakes on that. Do you really think that games are the only type of software that's feature, performance, and stability driven? That basically describes any piece of software that someone would try to sell to someone else. And it's not like the game industry has a monopoly on compressed schedules and death marches either.

[Rangoth]

No, the games industry does not have a monopoly on on compressed time scheduals or death marches. It does however have a very unique demand upon its programmers: to create at times nearly impossible acts of code under time frames that are as close to impossible as one can dream of. I got out of the industry for this very reason, its nuts man. There is no other industry that demands upon its best people slave hours. You are rarely granted the time to ENJOY learning the new skills. Is it different from programming for other industries? Hell yes! Other areas are granted a short break to sit back and enjoy what they have learned. You can think 'Man, thats pretty fuckin' cool right there. I can't believe I pulled that off.' You do that when working on a game during crunch time your going to get ground up and spit out of the door.

[chrisg]

Bollocks mate. By definition almost all game coding is bleeding fucking edge development under very difficult circumstances. If you think that is the same as curling out databasing solutions for a series of retail chains then you are sadly deluded.

The focus for game code is different than for regular code. Feature driven, performance driven, stability driven code is very hard to write to schedule and managing the correct balance means that game coders are under a LOT more pressure than most of their counterparts in other industries.

who talked about database solutions? Talk about database development / audio,video compression / embedded development / real time solutions and you have your bleeding edge right there. Others are just as much out there in R&D as the gaming industry is, just in other areas. And feature, performance and stability driven applies to any software that has to be sold as dolemite pointed out.

Eon, Rangoth and you talk about the pressure on the developers that makes the industry so different. Maybe the devs themselves should put an end on being exploited the way they are now.
A lot of the people in the game industry are highly skilled, yet they don't get together and demand what they deserve, both in terms of money and free time.

None of the big players can allow themselves a one year setback on a title because they had to fire the entire development team and higher a new one. That's the upside of the stockmarket and the shareholder value ;). Just because you have a passion for games and development does not mean you have to accept to be treated like a slave...

[Eon]

Well, the teams do it to themselves to a certain extent. Cutting features is something that games developers are notoriously bad at - and I don't think its a skill that gamers want us to develop.

Trust me, if you've never worked on a AAA title through to completion, you have no idea.

[Rangoth]

Trouble is right now there is a midset in the industry that anyone can be replaced. They do not realize that there is an art to making games, its not just writing a bunch of code on a screen.

[Xerxes]

It's not like it's going to be impossible... That's what dev kits are for right... Libraries and classes and such to help developers get sh** done...