For non-graphics related things, of course we have different test setups. I was talking about tests that specifically test graphics (and the code that interfaces with the graphics API).

One idea might be to have your simulation use a virtual clock. This clock will usually just output values that it gets directly from the real clock, and the simulation will run as normal. But in some circumstances (eg. while testing) you will ask the virtual clock to instead provide ‘fake’ clock data. This could be used during testing to make sure your simulation progresses by a known amount between frames.

This behaviour of the virtual clock can be exhaustively unit-tested, separately.

Obviously this has the problem the your program under test will not be running precisely the same code paths in testing as in production, but this may still be the simplest and most reliable way to address the issue.

Unless you can directly compare frames using a very simple image-comparison (eg. is every pixel identical), then I think image processing to compare sorta-similar frames is going to be hard and unreliable. I’d be tempted, for an application of this type, to only do minimal functional level testing by comparing images – just enough to verify that the program is running and framerates are sane and then test the rendering (of sky, ground, planes, shadows, etc.)

For the rest of the program behaviour that isn’t rendering-related, I’d then add a comprehensive layer of ‘high-level unit tests’, that exhaustively test, for example, plane movements by examining the changes to data structures in your program, or test the camera by examining the modelview matrix that is sent to OpenGL (or whatever), rather than examining the output from OpenGL.

I hope these ideas are applicable.

Best regards

I forgot a very important step:

Add a test for it. *Run the test to watch it fail*. Fix the problem. Then run the tests again to watch them pass.

All tests should be written this way – if the tests are written after the working code, you will be amazed how many times you write a test which has a bug in it and always passes. Running the test to see it fail helps ensure your tests don’t have bugs of this type.

You’re right that failures in totally unexpected ways are a problem.

I think the general testing philosophy to address this is that you create tests to cover as much as you reasonably can. Then every time you discover a new failure that the tests were not catching, you add a test for it. Fix it. Then run the tests again to watch them pass. In future you will be alerted if ever that particular failure regresses.

This isn’t perfect, but it’s still better than running with no tests at all.

Best regards.

Here is a more precise formulation of what I work on:

I work on creating a tool to test if 3D graphics scenes are correctly displayed. The idea is to store a setup that I know is correctly displayed and rendered as a reference scene. How this oracle scene is chosen to be the reference scene is not so important. Maybe it is verified manually.

The tool should be used on a military training simulator used to train Forward Air Controllers (FACTS). The simulation is written in C++ with Delta3D as engine. The tool might be integrated with a existing unit test framework.

On strategy that I currently work on, is to capture frames and compare them against the oracle using image processing algorithms. However this approach seems to be hard because how should the timing interval be selected so the tool works independent of the pc running the simulation. I mean performance issues of the underlaying hardware should not affect which frames that are captured.

Another approach could be to compare selected 3D models in the scene by traversing their scene graphs, and that way avoiding performance issues related to different CPUs and GPUs.

Hope to hear your ideas and thoughts towards this and if you have any references about something related I would appreciate it.

Best regards
Horn

i can see it being pretty hard to think of all the things that should be screenshot because some really random things can break. for example objects fading to the haze color when the camera looks a certain direction, corrupt geometry on character with > X bones, bad texture transform only on stage 2 or 3. i guess even these could be picked up by luck in a particular shot.

Ryan: thanks! It does not actually happen at the engine level, all that is required from the engine is the ability to do screenshots and load “game levels” (in this case it’s test scenes) in succession.

I work at a studio that used to contract for EA and was recently bought up, and I’ve wanted to do this for every game I’ve ever worked on. It’s much more cost-effective to do it at an engine level as you’re doing, and I’m so glad to hear that it actually works and is useful. I think I’ll keep around a bookmark to this entry to show my coworkers.

And FYI, EA has a compatibility testing lab for PC games that is quite rigorous. There’s always a bit of dread when submitting a build for compatibilty testing, because we invariably get strange bugs on the most obscure video cards.

That would save quite a bit of screenshot flipping.