Gabe Newell Worries about Next Gen

[Phades]

Next Generation (http://www.next-gen.biz/index.php?option=com_content&task=view&id=510&Itemid=2ration game titles," said Newell.) has an interesting article up where Gabe Newell, big boss of Valve, expresses his worries about the difficulties involved in making next generation games.

"Technologically, I think every game developer should be terrified of the next generation of processors. Your existing code, you can just throw it away. It's not going to be helpful in creating next generation game titles," said Newell.

"Most of the problems of getting these systems running on these multicore processors are not solved. They are doctoral theses, not known implementation problems. So it's not even clear that over the lifespan of these next generation systems that they will be solved problems. The amount of time it takes to get a good multicore engine running, the Xbox 360 might not even be on the market any longer. That should scare the crap out of everybody.

[Orphiuchus]

Uh-oh...

This is some fascinating stuff, and I'm glad to hear someone who everyone respects telling us what I've been saying for years: Microsoft and sony think we are all morons.

[crackeriah]

Meh.

"If you build it, they will come." Parallelism may present difficult problems, but the fact that PS3, X360 and future PCs are multiprocessor means that devs will figure out how to do something with those extra threads of execution. Devs that waste processor time will have inferior games compared to the competition.

[Theycallmeweezie]

Uh-oh...

This is some fascinating stuff, and I'm glad to hear someone who everyone respects telling us what I've been saying for years: Microsoft and sony think we are all morons.

I don't get it, so it must be true... how exactly do you mean?

[Bone]

The sky is falling!

[kickmybum]

Wow, if next-gen Phantom (woot!) simply drops a console with a Radion XT and a 3ghz processor, developers will flock to it and the games would probably look and play better too!

[president_fred]

I take it this means that the first generation of launch games on the new generation will be nothing graphically compared to later generations which is entirely different from every other console prior to this generation... oh wait, no, it's exactly the same. I think Newell has a point though, middleware that effectively utilises the hardware to great effect will become increasingly important to cut development costs and time.

[XenonCJ]

I have the utmost confidence that the John Carmacks of the world will figure out and utilize this stuff well... It's just a matter of time.

[Ernst_Jager]

Imagine how hard it will be for Cell programming. If Multi-core is this hard, that has to be a real bitch. With the 360 at least you somewhat have PC's in common.

[Kelegacy]

Technology may keep getting better every 6 months or so, but consoles dont always have to take that giant leap. They are producing games, not isotope scramblers. The only reason the Xbox 360 is coming out so soon is so that MS can try to squirt past Sony out of the gate. I think we might be entering new generations too quickly, and that hastiness will ultimately come back to bite the hardware manufacturers someday. Staying behind the curve tech-wise might benefit gamers, hardware corporations, and developers alike. The Xbox is still a very capable machine. It's a pity to see it be squashed in the dirt so quickly, especially when the games it produces are still beautiful and worthy.

It does scare me that Gabe thinks this. With development houses not able to keep up with the increasing costs to create for the next gen, this comes as more pressure on the bruise.

[StANTo]

Forking and piping multiple processes. The basics of C taught in colleges and universities alike. They really deem it that difficult? If it isn't that simple, there's something wrong with the interface they're putting forward to the programmers, surely?

Perhaps it's time for another level of coding language.

Considering 3d rendering packages have been abusing multi core CPUs for eons and also games such as Quake 3 and numerous nameless others have too, I don't think it's so much a development jump (from the glance I've seen anyway) as much a length of time to develop jump.

I'm sick of seeing realistic looking games anyway.

[Nath5000]

heh, gabe is just worried now that what he thought was a next gen engine like source (which he poured a lot of money into) is no longer good enough and has to be completely re-tooled for a multicore system.

[Dirty Harry]

Chill out if we can build it we can break it, if a engineer makes software encryption, someone will break it. I hope you get that all man made things generally are workable but this may bring people who specialize in three core programing and the such. So in someways it removes the fat from the industry and injects it with a need for these UBERprogramers?.

[Varsity]

heh, gabe is just worried now that what he thought was a next gen engine like source (which he poured a lot of money into) is no longer good enough and has to be completely re-tooled for a multicore system.
He said YOUR code. :p Source is entirely modular, so it's going to make Valve's transition to multi-core the simplest around. Replace the single-core stuff with everything else still intact and probably complete backward compatibility too.

[StANTo]

most of these problems happen in the cache

So what he means is, he's setup his forks and his pipes, yet he can't reference memory properly with pointers to be associated with the necessary alternate processing units.

No wonder he never touched Linux and Steam for linux runs terribly badly, considering Linux has been a multi processing OS for eons.

For those wondering what I'm on about:

http://en.wikipedia.org/wiki/Fork_%28disambiguation%29
http://en.wikipedia.org/wiki/Fork_%28operating_system%29
http://en.wikipedia.org/wiki/Computer_process

[XxSATANxX]

Saw HL2 on X-Box. Looked amazing!

Can't wait to own that game Steam Free!

[RMan]

These types of arguments are nothing new, and IMO are still just as silly as they've ever been. If they’re harder to develop for, then they’ll be harder to develop for across the board, effecting your competition as well (of course, some companies will get preferential treatment, giving them an edge). And even if it takes a great deal of money to make it work well for those systems, it’ll pale in comparison to the amount of money needed to hire the army of content creators needed at the top end of the next generation. IOW, these things are a problem, but I don’t see how they’re significantly worse than ever, and in the end the things that make the job harder will be defined primarily by the competition, the technology will not make a significant impact.

[Varsity]

So what he means is, he's setup his forks and his pipes, yet he can't reference memory properly with pointers to be associated with the necessary alternate processing units.
Funny, I thought it meant he and just about every other developer on the planet had single-core code and it wouldn't work multi-core.

[Kelegacy]

Chill out if we can build it we can break it, if a engineer makes software encryption, someone will break it. I hope you get that all man made things generally are workable but this may bring people who specialize in three core programing and the such. So in someways it removes the fat from the industry and injects it with a need for these UBERprogramers?.

Who would you consider FAT? Alot of development groups started out tiny (Larian, creators of Divine Divinity) and many places that produce quality RPG's for PS2, though not powerhouses like FF, are usually small. It always bothers me when a good firm goes out of business. Troika disappearing, despite their buggy tendencies, was a blow to the PC industry. Small houses are finding it harder and harder to survive in the console world. I dont just want hits like Burnout and companies like EA and Capcom throwing games at me; I want a diverse selection of blatant hits, cult-games like ICO, and an assortment of in-betweens. Shit like licensed stuff will always linger because EA usually releases it. We are setting ourselves up for expensive games, fewer developers and publishers, and a thinning herd of diversity.

As always, these are words from an eternal pessimist. Disregard what you will.

[StANTo]

Funny, I thought it meant he and just about every other developer on the planet had single-core code and it wouldn't work multi-core.

That was one of his points, yes.

[Varsity]

And you don't think that contradicts your dig a little?

[Dirty Harry]

Who would you consider FAT? Alot of development groups started out tiny (Larian, creators of Divine Divinity) and many places that produce quality RPG's for PS2, though not powerhouses like FF, are usually small. It always bothers me when a good firm goes out of business. Troika disappearing, despite their buggy tendencies, was a blow to the PC industry. Small houses are finding it harder and harder to survive in the console world. I dont just want hits like Burnout and companies like EA and Capcom throwing games at me; I want a diverse selection of blatant hits, cult-games like ICO, and an assortment of in-betweens. Shit like licensed stuff will always linger because EA usually releases it. We are setting ourselves up for expensive games, fewer developers and publishers, and a thinning herd of diversity.

As always, these are words from an eternal pessimist. Disregard what you will.
Whats wrong with capcom?

[Zeal]

He's just mad that everyone views Source as old news. Everyone's using the Unreal 3 engine.

And besides, developers are gonna have to get used to it, and that's why XNA was created in the first place. Multi-core everything is here to stay.

[Hellstorm]

Wow, a PC dev, a lazy one at that, whining that console development, where your games run best coded to the metal, is hard. Cry me a river. This isn't new, but then again I doubt Gabe ever had to code for the Saturn.

[Rogueski]

Like Gabe could code anyway.

And whoooo the new consoles have 25 times more power... so what... there will no doubt be a 2D streetfighter 2 2006 X alpha plus 3 squiggle and countless other top games that don't puish the console or need ten tonne of complex code... stop whinning and start thinking of sumat that isn't just Quake 2, again Gabe!

[bapenguin]

I think he's scared cuz Valve has a hard time with single core programming...how long was HL2 delayed? Man..HL3 will be at least 3 times as long for the delay.

[RMan]

He's just mad that everyone views Source as old news. Everyone's using the Unreal 3 engine.
Yea, but that's completely cyclic, since the same thing could have been said about Unreal at HL2’s release, when HL2/Valve had the best looking screenshots. I do think it’s a big mistake to take such an alarmist position to consoles, I doubt he’ll convince many people to avoid them. I think the fact that consoles would kill Steam has much to do with this position, likely more than the technical issues with getting Source working well on the consoles. In the end, I think he’s trying to get more people doing PC only (or titles that port to consoles after success) titles, since Source may be better suited for them than Unreal 3 or other console-centric engines they can get a greater foothold there. Still, I think this is likely a poor way to approach that goal, if that is their goal.

In the end they should have just worked towards getting it somewhat working on the consoles and pumped a good deal of HL money into producing some great content, they could have beat Unreal if they’d done that, since they definitely have some great IP to work with. Seems to me there’s a lack of high quality content coming from those guys, they dropped the ball in that respect.

[OSX]

Im not going to pretend I know anything about the technical side of this thread, and I really know this question is going to paint me as a fanboy, but didnt I read somewhere that Nintendo was making it so the current Gamcube API would be used for the Revolution as well? Or is this just text read off a website my head has made up.

[Draft]

Man's definitely sweating Source being outpaced by a bunch of newer, shinier engines. HL2 does not look good because of Source. HL2 looks good because the art if fucking phenomanal.

[Phades]

He may focus a bit more on the console side, but he's not trying to push more people to develop on PC's. Notice that he says "the next generation of processors," not consoles. Multi-core processors are also out now for PC's and are likely to become more and more common.

[RMan]

HL2 does not look good because of Source. HL2 looks good because the art if fucking phenomanal.
Yea, that was my point, that's why ANY engine looks good, and they should have focused more attention/money on it if they wanted to compete with Epic.

[RMan]

Notice that he says "the next generation of processors," not consoles. Multi-core processors are also out now for PC's and are likely to become more and more common.
Sure, but unlike consoles it'll take many years before it's dominant, and in the meantime the need for scalability will ensure that there will be far less concern with really utilizing them on the PC.

[Neverborne]

If they build it, there will be shovel-ware.

[Hellstorm]

Im not going to pretend I know anything about the technical side of this thread, and I really know this question is going to paint me as a fanboy, but didnt I read somewhere that Nintendo was making it so the current Gamcube API would be used for the Revolution as well? Or is this just text read off a website my head has made up.


This is true, you can use most of the GC API to make Rev games AFAIK, or more to the point that is what I heard. But that doesn't mean the Rev will be a single core CPU, although it could very well be. Heck I hear Nintendo is basically telling devs that if your game is just a port you can pretty much dev using the standard GC controller for your Rev version, as Nintendo still hasn't given out the secret to whatever the real Rev controller is.

[Hellstorm]

Man's definitely sweating Source being outpaced by a bunch of newer, shinier engines. HL2 does not look good because of Source. HL2 looks good because the art if fucking phenomanal.

Yes, I would tend to agree that art direction can go a long way in making your game look good. Take WoW for example, Blizzard has a killer art style that even with the low poly world of WoW makes the game look better than most games out there.

Unreal's art is fucking horrible, it's like 10 year olds trying to do Japanese cyborg-mechs and failing badly. Do they have to be that freaking bulky? Shit sometimes simple is better, something American devs never seem to grasp art wise.

Of course there is the so simple it's plain look, ie Halo.

[megatron666]

So what he means is, he's setup his forks and his pipes, yet he can't reference memory properly with pointers to be associated with the necessary alternate processing units.

No wonder he never touched Linux and Steam for linux runs terribly badly, considering Linux has been a multi processing OS for eons.

For those wondering what I'm on about:

http://en.wikipedia.org/wiki/Fork_%28disambiguation%29
http://en.wikipedia.org/wiki/Fork_%28operating_system%29
http://en.wikipedia.org/wiki/Computer_process

From Introduction to Parallel Computing, Second Edition (p. 46):
"The problem of keeping caches in multiprocessor systems coherent is significantly more complex than in uniprocessor systems. This is because in addition to multiple copies as in uniprocessor systems, there may also be multiple processors modifying these copies."

I could go on but maybe you should pick up the book and read the chapter. Also, you are talking about forks and pipes. This is fine if the system is just using heavy weight threads. But we don't know if this is true unless, of course, you have a dev kit.

No, I believe that they are going to go for light weight threads and use a shared memory architecture in the next gen. For that you will use shared memory and it makes development much harder. So I would say Gabe knows what he's talking about.

[Kamalot]

Yes, I would tend to agree that art direction can go a long way in making your game look good. Take WoW for example, Blizzard has a killer art style that even with the low poly world of WoW makes the game look better than most games out there.

Unreal's art is fucking horrible, it's like 10 year olds trying to do Japanese cyborg-mechs and failing badly. Do they have to be that freaking bulky? Shit sometimes simple is better, something American devs never seem to grasp art wise.

Of course there is the so simple it's plain look, ie Halo.

The world of HL2 has life. It feels like I am really there. The atmosphere and style and music are engrossing. While I love the action in Unreal Tournament the world feels static and lifeless. Anyone who says Half Life 2 is a bad looking game is looking at the specifications, not the art the engine creates.

[Venkman]

Sounds like the same whining we heard from the head of Oddworld when the PS2 came out. He is terrified of something new because he has so much invested in the previous generation of technology.

his statements lose a lot of credibility when people see PGR3 and Gears of War running in real time.

[51|RandoM]

NEWSFLASH:

"There has been another theft at Valve. Instead of taking code, the thieves made off with Gabe Newell's balls."

OK. until you figure out your doctoral thesis on writing multi-core code, just run on one core. Multi-core procs in every other platform are running the same old code written for single-core just fine, what is the big deal?

[SpectralWolf]

Aren't PGR3 and Gears of War using just one of the processor cores? AFAIK, a true multiprocessor engine is not present in the launch titles. Epic might be working on something there, but I don't know about anyone else working on one. Please correct me if I'm wrong.

[dojoteef]

Well as a Graduate student who majored in Computer Science and Math as an undergrad, I can definitely say Gabe knows what he's talking about. In order to fully utilize multiple processors to their limit, it takes specialized algorithms that are often counterintuitive to devise. If you've ever written a parallel algorithm, you'll fully appreciate that statement.

That does not even count synchronization issues, locking and unlocking threads, protecting data from changing during a computation, etc. Not only that, but the kicker is, debugging becomes a nightmare. That is not something that can just be overcome by being UBER. Gabe is fully correct in saying parallel algorithms are far from being a "solved" concept in programming.

Oh and one more thing, usually 2 processors don't actually give a doubling of speed for an application. Instead the best of the applicatons will go up by the root of the number processors. That means a 2 processor application usually peaks at about 1.414 times the speed of a single processor application.

Just food for thought.

[Vandenh]

He is right to a certain degree... but I am sure MS is working very hard to "upgrade" their compilers to do some of the hard work for you. 10 years ago I had to do a lot of optimizations myself to make games run at maximum speed, compilers now do this. It will be the same with multicore games dev and I am quite convinced that MS already has some killer tools in place. Then again... some things cannot be run multi-threaded so easily (especially if threads need to be depended on each other...).

[baz]

While I imagine fancy compilers will do a lot to minimise the in-order execution problems, they would have to be pretty fancy to automatically use mutliple processors. Although I did read somewhere that Hyundai were writing an auto-parallelizing compiler for the cell...

Saying crap like, duh you just need to fork your processes is pretty freaking dumb though. I guarentee gabe knows how to write multi-threaded code, he's just saying its a lot harder and that they are going to need better people, as well as re-visiting all the work that they have already done.

[Varsity]

You know, if someone from NCSoft had said this nobody would be arguing (although most of the derogatory posts here don't even qualify as arguments) at all.

[Vandenh]

>they would have to be pretty fancy to automatically use mutliple processors

Not really.. compilers today can already decide to run certain methods of different threads by analyzing the impact the method has on the program as a whole (minimal impact means easier to do). The big trick is, like someone said before, is to massively multi-thread your whole game/algorithm to work at peak efficiency. It is also a design problem... people will need to design their game "logic" around a hardware setup. Something that most games developers have never done before. So your games pipeline needs to be multi-core designed.

BTW "fork" has really not a lot to do with multi-cored CPU programming. It is a bit more involved :)

[Kentor]

He is right to a certain degree... but I am sure MS is working very hard to "upgrade" their compilers to do some of the hard work for you. 10 years ago I had to do a lot of optimizations myself to make games run at maximum speed, compilers now do this. It will be the same with multicore games dev and I am quite convinced that MS already has some killer tools in place. Then again... some things cannot be run multi-threaded so easily (especially if threads need to be depended on each other...).
It's not so simple anymore. This is not a case of knowing certain optimizations that have not yet been generalized for use in a compiler, but rather not knowing the behaviour of the system. It is no exageration on Newell's part when he states, "[m]ost of the problems of getting these systems running on these multicore processors are not solved."

In your example, the optimizations in question are almost purely implementation and mechanical optimizations. The issue at hand is that there are very few established techniques, or practices, for parallel SIMD computing since the field is quite new (SISD computing is equivalent to von Neumann architectures, while parallel SIMD systems have just come into existance in the past decade or so). In the world of parallel proccessing, many of the issues have not been solved even on a theoretical basis, while some of the problems have not even been demonstrated to be solvable.

With the new generation of hardware, a game developer doesn't just need to know sequential programming and the wealth of implementation details, but also at least a grasp of graph and set theory, computational theory, concurrency and more. To write production worthy fine grain parallel code you need at least a moderate understanding of many mathematics and computer science topics as well as practical knowledge: a feat that almost all computer science and engineering majors haven't accomplished by graduation.

This isn't some coarse grain parallel system using a bunch of SISD processors, we're talking about a shared memory space, shared cache, multiple SIMD core system. I doubt many game developers have even heard of PRAM algorithms, while most software developers have never even written a parallel algorithm in their lives.

dojoteef's comment is pretty much dead on.

[Zeal]

If you read about XNA, you'll see it was designed with multi-core processor support in mind. More and more software will emerge to efficiently program for systems with two or more processors.

Multiple cpus are necessary for next-generation software. This is why 360 and PS3 were designed as multi-core systems. It's that simple.

(I'm aware PS3 is using SPEs, but it's essentially the same concept)

[Speed_D]

The funny thing about Gabe's comment is that this problem already exists for PS2 developers. That architecture is a royal pain in the ass to program for. It is memory limited - that's its age showing. There are also quite a few gotchas to the sound and video architecture; certain things you just can't do at the same time and hang onto reasonable performance. The only reason PS2 games perform as well as they do is because the platform has been around for a long time now, and developers have learned its quirks.

I don't think developers are going to "solve" parallelism on the upcoming platforms, but they'll learn the particulars of each one over the span of a couple years. Mostly through trial and error.

[StANTo]

Amusingly the C and C++ modules at the University of Teesside for degree courses doesn't even touch on designing code for multiple processing systems. The HND (a relative lesser qualification) on Computing goes into crazily greater detail.

A lot of students coming out now, from Computing degrees of any form haven't even touched multiple processing code writing. That just makes me chuckle, and is just going to make it overall more difficult for companies to take advantage of.

[dojoteef]

If you read about XNA, you'll see it was designed with multi-core processor support in mind. More and more software will emerge to efficiently program for systems with two or more processors.
The problem is there is no theoretical basis that proves this. Mathematicians and Computer Scientists have been working on the concept of parallel processing for a while now and this is what one of my Algorithms books has to say about it:

"The differences between the various existing parallel machines, even as far as the naive use is concerned, are major. We can no loner adopt one 'generic' model of computation and hope that it adapts to all parallel computers. Designing parallel algorithms, analyzing them, and proving them correct is much more difficult than the corresponding steps for sequential algorithms."

These are people who look at the theoretical abilities of computers and they claim it's a hard problem. I trust them, especially since I've had experience having to design parallel algorithms in undergrad.



Multiple cpus are necessary for next-generation software. This is why 360 and PS3 were designed as multi-core systems. It's that simple.
Multi-core processors are definitely not "necessary". Don't buy into the hype. I wouldn't be surprised if the Nintendo Revolution is a single core processor.


I'm not saying multi-core processors should never be used, I just think it's too early to expect game developers to actually be able to deal with such systems efficiently given the short development cycles they have.

I do think Sony took a smarter approach by having the 7SPEs, because they are essentially vector processors. They should be usable as rigid body physics processors, similar to the concept of the GPU. Some of the problems though are that their is a shared memory structure for the SPEs and few applications are truly vectorizable. Graphics and physics have been shown to be, but general game logic, AI, sound, networking, etc haven't been shown to be vectorizable.

Ultimately I'm saying don't bitch at Gabe so much; he's not just blowing smoke up your ass.

[Kentor]

Mathematicians and Computer Scientists have been working on the concept of parallel processing for a while now and this is what one of my Algorithms books has to say about it:
Just curious, but which algorithms text book is the excerpt from?

[Wonka]

The industry has changed in a major way. All the major bottlenecks and challenges for the next gen of consoles are software challenges. Multiple processors, online services etc. In contrast, hardware will likely not be that big of a factor this time around (odds are good that even the next Nintendo console will be more than powerful enough to make a very good game in almost any genre).

So lets see, which of the "big three" has a lot of expertise in writing good software???

[dojoteef]

"Introduction to Algorithms: A Creative Approach (http://www.amazon.com/exec/obidos/tg/detail/-/0201120372/103-0401593-5090222?v=glance)" by Udi Manber (http://manber.com/). He is now Chief Algorithms Officer for Amazon.com, whatever that title means.

[bobbler]

It seems to me that Gabe Newell is overplaying it a bit. There are plenty of issues that you come across when you try to split tasks that are inherently serial, however, there are also several tasks which lend themselves well to being parallel (with a bit of fighting). You'll break your back trying to split everything into several threads -- some things just aren't worth the trouble. The game thread and such can be split into 2-3 different threads without much sweat and then with a bit more sweat things like AI and Physics can be split further -- other things like sound will take up a thread since there is no sound hardware on either PS3/Xbox360 -- then we have things like precedural geometry and textures can take a thread and be happy. Console developers (especially those on PS2) have had to deal with some crazy things in the past and they've, for the most part, shown they can handle whatever Sony/MS/Nintendo throw at them hardware wise.

I wouldn't say that we won't see anything that uses fully threaded engines until the end of the consoles lifetime (which, even if that was the case, PS4/Xbox3/NS6 era will inevitably have multiple cores also -- and PCs will as well, so the effort isn't in vain, the industry is moving towards it no matter how much people don't want to deal with it). We are in the era where ILC (OOO, Superscaler, complex branch predictors, etc) has, for the most part, ceased to be king and TLP (SMT and SMP) has the new crown. Intel and AMD saw that they are reaching some poor returns on their ILC designs for how much effort they had to put forth (die space) to extract more ILP/IPC -- meanwhile they could just paste on an extra core and theoretically double the performance, which is nice especially in marketing (of course it all depends on the situation -- running multiple programs vs running a multithreaded program).

With the consoles developers have to deal with weak ILP and strong TLP, which is pretty much the opposite extreme of what we had in the PC space, exclusively, only 2 years ago (single core A64/P4). However, going from previous generation of consoles the difference isn't all that huge -- both Xbox360 and PS3 are more akin to a PS2 than a PC (or Xbox1). There are growing pains for developers that went from PC to PS3/Xbox360 or Xbox to PS3/Xbox360 -- developers that had to deal with PS2 are probably better off this generation (the vu0, vu1, simple core and then the GS).

I wouldn't be too concerned that we won't get every last bit of performance that can feasibly be taken out of the consoles by the end of the generation. Consoles are vastly different from PCs, you have a closed box that never changes so you know the hardware and can dig deep to get there if need be -- with PCs are you dealing with APIs and Drivers at best, not the hardware.

[Kentor]

"Introduction to Algorithms: A Creative Approach (http://www.amazon.com/exec/obidos/tg/detail/-/0201120372/103-0401593-5090222?v=glance)" by Udi Manber (http://manber.com/). He is now Chief Algorithms Officer for Amazon.com, whatever that title means.
Ah... thought it might have been from Introduction to Algorithms by Cormen, Leiserson, Rivest (http://theory.lcs.mit.edu/~clr/).