Compiling HLSL into GLSL in 2010
Realtime shader languages these days have settled down into two camps: HLSL (or Cg, which for all practical reasons is the same) and GLSL (or GLSL ES, which is sufficiently similar). HLSL/Cg is used by Direct3D and the big consoles (Xbox 360, PS3). GLSL/ES is used by OpenGL and pretty much all modern mobile platforms (iPhone, Android, …).
Since shaders are more or less “assets”, having two different languages to deal with is not very nice. What, I’m supposed to write my shader twice just to support both (for example) D3D and iPad? You would think in 2010, almost a decade since high level realtime shader languages have appeared, this problem would be solved… but it isn’t!
In upcoming Unity 3.0, we’re going to have OpenGL ES 2.0 for mobile platforms, where GLSL ES is the only option to write shaders in. However, almost all other platforms (Windows, 360, PS3) need HLSL/Cg.
I tried a bit making Cg spit out GLSL code. In theory it can, and I read somewhere that id uses it for OpenGL backend for Rage… But I just couldn’t make it work. What’s possible for John apparently is not possible for mere mortals.
Then I looked at ATI’s HLSL2GLSL. That did produce GLSL shaders that were not absolutely horrible. So I started using it, and (surprise!) quickly ran into small issues here and there. Too bad development of the library stopped around 2006… on the plus side, it’s open source!
So I just forked it. Here it is: http://code.google.com/p/hlsl2glslfork/ (commit log here). There are no prebuilt binaries or source drops right now, just a Mercurial repository. BSD license. Patches welcome.
Note on the codebase: I don’t particularly like the codebase. It seems somewhat over-engineered code, that was probably taken from reference GLSL parser that 3DLabs once did, and adapted to parse HLSL and spit out GLSL. There are pieces of code that are unused, unfinished or duplicated. Judging from comments, some pieces of code have been in the hands of 3DLabs, ATI and NVIDIA (what good can come out of that?!). However, it works, and that’s the most important trait any code can have.
Note on the preprocessor: I bumped into some preprocessor issues that couldn’t be easily fixed without first understanding someone else’s ancient code and then changing it significantly. Fortunately, Ryan Gordon’s project, MojoShader, happens to have preprocessor that very closely emulates HLSL’s one (including various quirks). So I’m using that to preprocess any source before passing it down to HLSL2GLSL. Kudos to Ryan!
Side note on MojoShader: Ryan is also working on HLSL->GLSL cross compiler in MojoShader. I like that codebase much more; will certainly try it out once it’s somewhat ready.
You can never have enough notes: Google’s ANGLE project (running OpenGL ES 2.0 on top of Direct3D runtime+drivers) seems to be working on the opposite tool. For obvious reasons, they need to take GLSL ES shaders and produce D3D compatible shaders (HLSL or shader assembly/bytecode). The project seems to be moving fast; and if one day we’ll decide to default to GLSL as shader language in Unity, I’ll know where to look for a translator into HLSL :)
Actually, I’ve been able to get Cg to produce working GLSL. I’m curious what went wrong with yours. Did you make sure to use cgCombinePrograms()?
I want the compilation to happen purely offline. With Cg’s GLSL translator, it seems to be designed to be used at runtime (so you’d use Cg runtime functions to manage programs, set uniforms etc.). I don’t want to ship Cg runtime in my runtime :) (and Cg runtime on some platforms does not exist, e.g. iPhone or Android).
Were you doing Cg to GLSL cross-compilation offline, without using the runtime?
No, right now I am doing the compilation at runtime. Currently, I am writing my own renderer so I can experiment with various rendering techniques to test their merit. Since my focus is deferred rendering, I don’t have a lot of shader combinations. So shader compilation at runtime hasn’t really been an issue thus far.
Ultimately, I guess my suggestion isn’t applicable to your situation.
I dont understand why an engine like Unity with roots in the Mac/OS should use HLSL at all.
Why not use GLSL for everything and port to HLSL only for XBox.
@Dimi: because GLSL sucks? :)
Seriously though, if we put mobile platforms aside, here’s what we use in Unity in 2010:
* Windows: D3D9 (assembly shaders, compiled from HLSL)
* Mac: OpenGL (ARB assembly shaders, compiled from Cg)
* Xbox 360: HLSL is the only option
* PlayStation 3: Cg is the only option
See? None of the platforms, except mobile platforms with OpenGL ES 2.0 use GLSL. If GLES2.0 would have another option for shaders, we’d gladly use it.
Why we aren’t using GLSL on OpenGL (Mac)? Because for a very long time, it was unstable as jello. Mysterious bugs in GLSL parser/optimizer, unpredictable runtime behavior, etc. There’s just a lot less hidden pitfalls when using ARB assembly shaders. Maybe for Unity 3.0 we’ll switch to GLSL on Mac by default, but that is still up in the air.
I was able to do exactly what you’re trying to achieve. For my home project I have a number of cookers, each of them for a single platform (similar to yours, except I prefer GLSL over ARB). During a processing time (can be offline) the cooker loads a HLSL shader using Cg, process it and submit to a library that saves shaders in a unified file taking care of endianess and stuff. I’ve been using FX file format for my shaders, mostly as it is simple to manage and good enough to convert to various platforms (using given API or Cg). So, the GLSL cooker uses cgCreateEffectFromFile then iterates thru its techniques, passes and parameters getting all the important data to speed up loading time and finally get the shaders code using cgGetProgramString. The GLSL code looks a little weird but it works. However I am still fighting with getting vertex attributes instead of gl_Vertex & Co. Drop me an email if need more details. Cheers!