xStream snippets

[Varsity]

Some new information has come to light (http://dukenukem.typepad.com/game_matters/2005/06/digital_distrib.html) about the digital distribution xStream service, which was recently chosen by 3D Realms as the online platform of choice for Duke Nukem: Forever and Prey.

Streaming also works with game patches, allowing a game to be patched while you are playing it, such that it is entirely transparent and hassle-free. Wouldn't this be a great feature for those of us who almost weekly have to put up with WoW patches that can take 30 - 120 minutes to patch, keeping us from playing the game while the patch is in progress?Not all that interesting until you realise that this includes code - quite how I can't begin to imagine, but Miller made it clear in a later comment. That's cool, but it can't top this:
Applying the technology to the master files can even turn the CD or Gold Master into a streaming device. Turning a PC into a console. This means that the Game xStream technology can turn any retail sold CD into a Tray-n-Play CD. Imagine putting in a game (CD) and playing it within seconds, that would normally require upwards of 45 minutes to install!And all this without the game's developers lifting a finger. How can anyone say no?

Yeah, so I'm assuming Manofgod works for DiStream. I think it's a safe enough bet given what he said.

[KarmaGhost]

It's "xStream," right? The title says it's "xSteam." Yeah, proofreading is good.

[Deadend]

So... does this mean 3D Realms is switching engines... again? I have no clue how they can have this tech work around a standard engine with very little overhead.

[holysin]

"Imagine putting in a game (CD) and playing it within seconds, that would normally require upwards of 45 minutes to install!"

The only game I can remember that takes forever to install is WoW. Other than that, most games install pretty fast.
I'd rather take 5-10 min to install than have to deal with longer load times.

[XxSATANxX]

HL2 took 2 hours to install BECAUSE OF STEAM:

[Praetor-Vong]

LOL
Yeah, I hated the long install time for HL2. It rather sucked Hard Core. This xStream sounds interesting to say the least. Of course, it's success is dependant on many factors. I can deal with longer load times for the first hour of a game while playing it. It sounds like a good trade-off.

[Rommel]

The only game I can remember that takes forever to install is WoW.

Ever play Final Fantasy XI?

[Rone]

Does WoW really take from 30 minutes to 2 hours just to patch? That's hideous.

[mos]

Yup, watching a buddy install FFXI on his machine was torture, even for me.

[GrinR]

Xthats Xso Xcool Xthey Xare Xnaming Xit Xstream Xbecause Xthat Xmakes Xit Xseem Xso Xmuch XCOOLER!!

[H.M._Murdock]

They really could not come up with a name that is not contain the name of the product they are copying?

[jeffool]

Granted something like "GameStream" may have been a cooler name, but screw the name, this is technologically kick-ass if it works well and wide.

[Bones4lyf]

Do any of you guys know when Uwe Boll's gonna make a tetris movie?

[Rone]

Right after John Woo finishes the Katamari Damacy movie (http://www.gamespot.com/news/2005/04/01/screens_6121444.html?page=1).

[Varsity]

It's "xStream," right? The title says it's "xSteam." Yeah, proofreading is good.
It was late. :p

So... does this mean 3D Realms is switching engines... again? I have no clue how they can have this tech work around a standard engine with very little overhead.Nor me, but it does.

[Qoz]

The blog says that no time is spent by the developers putting game data into packages etc. How is this possible? How will the system know what data to fetch first? It's hard to believe that games don't need some extra coding also - like when data is not available to proceed etc. It's seems like a good idea for linear games, but what about GTA:san andreas? You can quickly drive through ALOT of city-parts (textures etc.) right away.
How can I play Doom3 or other huge games with large datafiles?
I find it hard to believe, that no developing time is needed to make this work.
But then again the concept of buying and playing almost right away would attract alot of impulse buyers (like he said).

[Kentor]

So... does this mean 3D Realms is switching engines... again? I have no clue how they can have this tech work around a standard engine with very little overhead.
Here's a random guess -- no manual engine changes required at all:
They probably are using an API that sits between all the game resources and the engine (i.e. the system replaces your own read resource with their own calls). The API probably allows the game to run in a local resource mode and a streamed resource mode. If I were to do it, I would write a preprocessor (think C++ preprocessor) that looks through the source for all the resource load calls and replaces the calls with their own while building a manifest of all the resources (determine what resources are loaded and of what type). This manifest could then be used to build a segmented resource bundle. You could determine which resource segments need to be loaded first via a profile that could be automatically generated by just running the game (i.e. run the game after preprocessing and the sequence of resource loads could then be recorded). The profile could be tweaked manually as well to allow for certain resources to explicitly loaded early, and for marking signficant resource load points if they could not be determined automatically (i.e. minimal resources for game start, minimal resources for first level, etc).

From that point, streaming the resources to get the game running within a few minutes of a download shouldn't be too difficult (unless say your game starts with a 45 minute HD video)... just have the streaming client load the manifest and profile, then download the resource segments in the required sequence. Once sufficient resources for the game to start playing have been loaded, start the game.

There's probably a lot of gritty details like how much lead time needs to be given before a game should be started so you won't have the game stall, determining the optimal sequence of segment loads, what segments should be directly streamed on request (i.e. videos), what subsegments are loaded first (i.e. you have a very large resource file that loads certain offsets first), etc., but the basic solution is probably something along the lines above.

Steam does uses a similar technique in terms of the API (i.e. they use their own proprietary file system calls which access resouce bundles), but for some reason the whole streaming the bundles idea elluded them.... I was rather surprised when HL2 came out and Steam didn't implement something along the lines of xStream.

On the subject of live patching, I'm guessing they just mean the game can be patched while you're playing the game... not that the game you're playing will suddenly have bug fixes and new features appear while you're playing. (i.e. while you're playing the patch is streamed to your client and is applied, but the changes will only take effect upon shutting down the game and starting it again). So their solution is probably just a management system that keeps multiple versions of the game and resources on the machine at once so that a game can be played while a patch is not completed.

[Varsity]

Kentor, you might find C:\WINDOWS\system32\drivers\DiGameHook.sys interesting. ;)

[Kentor]

Kentor, you might find C:\WINDOWS\system32\drivers\DiGameHook.sys interesting. ;)
So is then the implication they're using a file system hook or driver extension instead (i.e. you could use it on "out of box" games)? Hmmm... that seems somehow less versatile, but I suppose it's more elegant in some fashions. At least you don't need to do a rebuild.

It still doesn't really answer how they would go about optimizing the load sequence without generating sometype of resource load profile first (or having the load profile provided). Functionally, hooking into the file system seems approximately the same as replacing the file system calls pre-build, but you get the advantage of working against "out of box" games, but again, there wouldn't be much information to work against without sometype of load profile. I guess you could do something like paging in file blocks prioritizing using the sametype of algorithms, but still... obtaining that solid load sequence seems to be what would make or break the system.

[Varsity]

I imagine games still need profiling, it's just done by xStream and not the devs. They could build up a huge library of streaming profiles for games they don't officially support and have them downloaded as they are called on.

[Mason]

There's probably a lot of gritty details like how much lead time needs to be given before a game should be started so you won't have the game stall, determining the optimal sequence of segment loads, what segments should be directly streamed on request (i.e. videos), what subsegments are loaded first (i.e. you have a very large resource file that loads certain offsets first), etc., but the basic solution is probably something along the lines above.

Steam does uses a similar technique in terms of the API (i.e. they use their own proprietary file system calls which access resouce bundles), but for some reason the whole streaming the bundles idea elluded them.... I was rather surprised when HL2 came out and Steam didn't implement something along the lines of xStream.

On the subject of live patching, I'm guessing they just mean the game can be patched while you're playing the game... not that the game you're playing will suddenly have bug fixes and new features appear while you're playing. (i.e. while you're playing the patch is streamed to your client and is applied, but the changes will only take effect upon shutting down the game and starting it again). So their solution is probably just a management system that keeps multiple versions of the game and resources on the machine at once so that a game can be played while a patch is not completed.

Also, recall that those "gritty details" would vary greatly from system to system. The resolution of the textures and models, the shaders involved, the optical media and network connection speed, all of these would have impact on how the streaming would work. And in the end, you're either 1) blocking the player with a load screen until all resources are loaded, or 2) letting the player enter an incomplete level and play until the resources are loaded.

If #1 is the case, then streaming is of limited advantage to anyone. You could do small updates (fix a texture or something) fairly well, but no one wants to have play interrupted by five-minute load screens.

#2 can work okay if you do it intelligently, i.e. force the acquisition of the resouces for a given segment of a level, and then stream in resources beyond a certain range. That'd take a crap-ton of manual profiling to do in a non-annoying way. And if players frequently run into texture-less areas or Attack of the White Cube scenarios, I'd be willing to bet they'd far rather just install off a DVD for a few minutes and enjoy a flawless play experience.

And in any case, you'd need a huge network of content servers to pull it off. That's expense for a service of questionable value to anyone, for a single-player game.

Guild Wars uses technology similar to this, and it does a wonderful job. But GW is an ever-evolving world that the devs are constantly working on, as expansion of the game world is a core part of ArenaNet's business model. Canonical shooters like DNF just really don't need this technology, and I find it highly unlikely that their dev teams would make much use of it post-launch. 3D Realms is just determined to whip out a bigger ePeen than Valve.

And regarding live patching, yeah they're giving a misleading impression there. Now, it *is* possible to do something much like this in Win32. Basically, keeping most parts of game functionality exported in DLLs, and then shadow-copying the game libraries prior to execution. Whenever there's an update downloaded, you just replace the library in the game directory, shadow-copy the new file, and route all library calls to the new library. This would add some overhead, though, due to the need to manage all library calls. It could be a pretty thin layer, but still, overhead all the same.

You still wouldn't be able to update the running executable, without working out a painfully complex method of launching a new process and sending over all of the game state information between processes. IPC being what it is, nobody's dumb enough to waste time on this. But with a trivial executable and a good shadow-copy scheme for the libraries, you could effectively make game updates on the fly.

[Mason]

I imagine games still need profiling, it's just done by xStream and not the devs. They could build up a huge library of streaming profiles for games they don't officially support and have them downloaded as they are called on.

This statement is perhaps confusing three concepts. As near as I can tell, xStream involves these functions: digital distribution of game content, streaming updates, and (possibly) streaming from optical media to avoid installs.

Digital distribution can be seen in the conventional sense (Steam-esque downloading of files, you can play the game when they're all downloaded) or a more ambitious system that lets you play almost immediately and streams content down. Almost any game could handle conventional distribution without developer support, but streaming-installs are very different.

First of all, depending on how the game is set up, a streaming install might have almost zero purpose. The aforementioned intro movies are a good example. Also, a game like HL2 uses a very large chunk of its total resources just to render the first scene of the game. You'd need all the city 17 textures, most of the character models, sounds, etc just to start things off. It isn't like streaming a video in this sense, it isn't linear. And if you have to wait for the game to be 70% downloaded before you can play at all (and then have long load times whenever the player catches up with the streaming), it makes a streaming install seem pretty worthless. Why not just wait for 100%, only a marginal increase in time?

Second, not all developers play nice. If someone writes low-level file system IO into their game, or gets overly-paranoid (like checking a resource manifest to make sure certain files exist before running), then a streaming scheme wouldn't work without taking complete control of a player's IO subsystem. People might get leery of that. Many would probably rather haul their asses to the store rather than let a 3rd-party app reinterpret all of their file system calls and download content as it sees fit. Or just use a conventional online distribution method that avoids all of this nonsense.

Streaming updates obviously have no point without developer support. xStream ain't going to tweak your textures for you. And must I get that new texture five minutes after the developer gets done messing with it, or can I wait for an aggregate patch that's been through extensive QA? It really depends.

And using a PC like a console is very specialized, not many games would do this well. I mean, okay you could stream data off the WoW CDs, but who the hell wants to keep changing CDs in order to play WoW? Honestly, smart installation programs are more necessary than scrapping installation altogether. And most players would rather browse forums for 10 minutes while a game installs, than play instantly and with substantial load times (which are a given with optical media).

So there are a few things that xStream could do without developer support, but none of them are too amazing, and most have serious drawbacks. A really solid platform for conventional online game distribution is what the industry truly needs. Hell, using bittorrent to get games doesn't involve streaming, yet gamers seem really fond of the practice.

[Bones4lyf]

WOOT!!! duke nukem forever is coming o........wait.....DAMN IT.
"yeah i know over used joke"

xStream is to duke nukem and prey as half-life is to everyone's favorite gaming service "steam"...yeah i hope xStream gets this right becouse i dont want to wait 4 hours just to install one game *cought* half-life 2 *cought*

[Kentor]

Also, recall that those "gritty details" would vary greatly from system to system. The resolution of the textures and models, the shaders involved, the optical media and network connection speed, all of these would have impact on how the streaming would work. And in the end, you're either 1) blocking the player with a load screen until all resources are loaded, or 2) letting the player enter an incomplete level and play until the resources are loaded.
Well... they wouldn't impact how it would work, but it would impact how well it would work (what does optical media have to do with this at all btw?). I have serious doubts as to how well their system will work on games that require massive resources to render a single scene that varies with time quickly. There doesn't seem to be much information on their site as to how they'll deal with the data starvation situation (e.g. what you're describing).

#2 can work okay if you do it intelligently, i.e. force the acquisition of the resouces for a given segment of a level, and then stream in resources beyond a certain range. That'd take a crap-ton of manual profiling to do in a non-annoying way. And if players frequently run into texture-less areas or Attack of the White Cube scenarios, I'd be willing to bet they'd far rather just install off a DVD for a few minutes and enjoy a flawless play experience.
The profile is the crux of the problem. I don't think that it would require that much manual profiling though... not if you did it intelligently. Consider dynamic branch predicition... why doesn't branch predicition require a massive amount of profiling to become quite accurate? Or memory paging and caching algorithms... why are they accurate at all without having explicit knowledge? I do think the profile is the tough problem here, but it does seem surmountable.

And in any case, you'd need a huge network of content servers to pull it off. That's expense for a service of questionable value to anyone, for a single-player game.
I tend to agree that it isn't of much use for the initial game release, but it does seem quite useful for things like expansion packs. Servers are not an issue at all though... buying a cluster of a hundred, or even a couple hundred, content servers is not a problem (or if it is, you should probably be thinking of other problems than throwing money at making a game load a few minutes faster). Even server bandwidth isn't really much of an issue... I think the problem is that the last mile bandwidth today isn't really upto the job in many cases. Consistent high bandwidth in the last mile is not really that common in North America.

Guild Wars uses technology similar to this, and it does a wonderful job. But GW is an ever-evolving world that the devs are constantly working on, as expansion of the game world is a core part of ArenaNet's business model. Canonical shooters like DNF just really don't need this technology, and I find it highly unlikely that their dev teams would make much use of it post-launch. 3D Realms is just determined to whip out a bigger ePeen than Valve.
I'm not sure about that... I bet this will be useful for things like expansion packs and such (where the amount of data is relatively small as would be the case in GW patches). I agree though: I don't see this as being very useful on day one when they release DNF.

You still wouldn't be able to update the running executable, without working out a painfully complex method of launching a new process and sending over all of the game state information between processes. IPC being what it is, nobody's dumb enough to waste time on this. But with a trivial executable and a good shadow-copy scheme for the libraries, you could effectively make game updates on the fly.
I don't find IPC complicated (dunno... seems pretty simple after using it for awhile)... but I would question whether it was worth the time and effort making hot patches versus background patches.

[Kentor]

Second, not all developers play nice. If someone writes low-level file system IO into their game, or gets overly-paranoid (like checking a resource manifest to make sure certain files exist before running), then a streaming scheme wouldn't work without taking complete control of a player's IO subsystem.
Say what? What's the problem with xStream just replacing stdio or just hooking into the file system services? How would you, or a game developer, bypass a kernel hook exactly (i.e. show me some code that demonstrates this)?

EDIT
-should have read "...you, or..."

[Varsity]

Mason...what on earth?

This statement is perhaps confusing three concepts. As near as I can tell, xStream involves these functions: digital distribution of game content, streaming updates, and (possibly) streaming from optical media to avoid installs.They are all (in this case at least) the same thing. Games are distributed either from a server or a CD and can be played before the download/copy is complete.

First of all, depending on how the game is set up, a streaming install might have almost zero purpose.Yes...but so what?

The aforementioned intro movies are a good example. Also, a game like HL2 uses a very large chunk of its total resources just to render the first scene of the game. You'd need all the city 17 textures, most of the character models, sounds, etc just to start things off.Again yes, but why is this a problem? So perhaps it takes longer to 'buffer', it doesn't mean streaming is a waste of time.

It isn't like streaming a video in this sense, it isn't linear.Depends on the game. HL2 and Doom3 ARE linear, so content streaming works nicely.

Why not just wait for 100%, only a marginal increase in time?Because it is an effortless timesaver? Seriously, these are all things that make streaming imperfect but none of them come close to making it a waste. In this worst-case scenario I get to play the game 30% earlier than you. That's still a sizable advantage with no drawbacks.

Second, not all developers play nice. If someone writes low-level file system IO into their game, or gets overly-paranoid (like checking a resource manifest to make sure certain files exist before running), then a streaming scheme wouldn't work without taking complete control of a player's IO subsystem.So don't stream those games...

Streaming updates obviously have no point without developer support. xStream ain't going to tweak your textures for you. And must I get that new texture five minutes after the developer gets done messing with it, or can I wait for an aggregate patch that's been through extensive QA? It really depends.Who says it has to be like that? What stops anyone from the simple, logical and sane solution to that problem of waiting for a patch to be finished first? Are you feeling OK? ;)

And using a PC like a console is very specialized, not many games would do this well. I mean, okay you could stream data off the WoW CDs, but who the hell wants to keep changing CDs in order to play WoW?I hope at this point you can guess what is coming next: don't use the system on games with multiple disks.

And most players would rather browse forums for 10 minutes while a game installs, than play instantly and with substantial load times (which are a given with optical media).Do you know that, or is it just what you personally think?

I really don't know what you are trying to achieve here. Streaming isn't the perfect solution to everything that will cure AIDs and feed your kittens, as you have realised, but you then seem to draw that out to impossible lengths to make the claim that it is next to useless?