Comments on: Testing graphics code http://aras-p.info/blog/2007/07/31/testing-graphics-code/ Random thoughts of a triangle pusher Thu, 20 Nov 2008 23:10:28 +0000 http://wordpress.org/?v=2.6.3 By: Game Rendering » Regression Testing a Renderer http://aras-p.info/blog/2007/07/31/testing-graphics-code/#comment-14479 Game Rendering » Regression Testing a Renderer Fri, 07 Nov 2008 11:46:40 +0000 http://aras-p.info/blog/2007/07/31/testing-graphics-code/#comment-14479 [...] Information about how Unity does testing of their graphics code http://aras-p.info/blog/2007/07/31/testing-graphics-code/ [...] [...] Information about how Unity does testing of their graphics code http://aras-p.info/blog/2007/07/31/testing-graphics-code/ [...]

]]> By: realtimecollisiondetection.net - the blog » A brief graphics blog summary http://aras-p.info/blog/2007/07/31/testing-graphics-code/#comment-11389 realtimecollisiondetection.net - the blog » A brief graphics blog summary Mon, 14 Apr 2008 07:29:24 +0000 http://aras-p.info/blog/2007/07/31/testing-graphics-code/#comment-11389 [...] not have noticed gamma being broken for several weeks (if indeed ever).  Another good post on testing graphics code can be found at Aras Pranckevičius’ [...] [...] not have noticed gamma being broken for several weeks (if indeed ever).  Another good post on testing graphics code can be found at Aras Pranckevičius’ [...]

]]> By: Horn http://aras-p.info/blog/2007/07/31/testing-graphics-code/#comment-11030 Horn Mon, 17 Mar 2008 10:59:49 +0000 http://aras-p.info/blog/2007/07/31/testing-graphics-code/#comment-11030 I just came across your blog and it seems that you are the right person to ask about Test Driven Development and Unit Testing in 3D graphics. I am doing a thesis in applying Test Driven Development in 3D training simulations, but I have trouble finding relevant material about this. Here is a more precise formulation of what I work on: I work on creating a tool to test if 3D graphics scenes are correctly displayed. The idea is to store a setup that I know is correctly displayed and rendered as a reference scene. How this oracle scene is chosen to be the reference scene is not so important. Maybe it is verified manually. The tool should be used on a military training simulator used to train Forward Air Controllers (FACTS). The simulation is written in C++ with Delta3D as engine. The tool might be integrated with a existing unit test framework. On strategy that I currently work on, is to capture frames and compare them against the oracle using image processing algorithms. However this approach seems to be hard because how should the timing interval be selected so the tool works independent of the pc running the simulation. I mean performance issues of the underlaying hardware should not affect which frames that are captured. Another approach could be to compare selected 3D models in the scene by traversing their scene graphs, and that way avoiding performance issues related to different CPUs and GPUs. Hope to hear your ideas and thoughts towards this and if you have any references about something related I would appreciate it. Best regards Horn I just came across your blog and it seems that you are the right person to ask about Test Driven Development and Unit Testing in 3D graphics. I am doing a thesis in applying Test Driven Development in 3D training simulations, but I have trouble finding relevant material about this.

Here is a more precise formulation of what I work on:

I work on creating a tool to test if 3D graphics scenes are correctly displayed. The idea is to store a setup that I know is correctly displayed and rendered as a reference scene. How this oracle scene is chosen to be the reference scene is not so important. Maybe it is verified manually.

The tool should be used on a military training simulator used to train Forward Air Controllers (FACTS). The simulation is written in C++ with Delta3D as engine. The tool might be integrated with a existing unit test framework.

On strategy that I currently work on, is to capture frames and compare them against the oracle using image processing algorithms. However this approach seems to be hard because how should the timing interval be selected so the tool works independent of the pc running the simulation. I mean performance issues of the underlaying hardware should not affect which frames that are captured.

Another approach could be to compare selected 3D models in the scene by traversing their scene graphs, and that way avoiding performance issues related to different CPUs and GPUs.

Hope to hear your ideas and thoughts towards this and if you have any references about something related I would appreciate it.

Best regards
Horn

]]> By: blackpawn http://aras-p.info/blog/2007/07/31/testing-graphics-code/#comment-1522 blackpawn Sat, 25 Aug 2007 07:23:21 +0000 http://aras-p.info/blog/2007/07/31/testing-graphics-code/#comment-1522 whoa hardcore. i'm still doing things pretty ad hoc and relying a lot on alpha testers to report problems. i can see it being pretty hard to think of all the things that should be screenshot because some really random things can break. for example objects fading to the haze color when the camera looks a certain direction, corrupt geometry on character with > X bones, bad texture transform only on stage 2 or 3. i guess even these could be picked up by luck in a particular shot. whoa hardcore. i’m still doing things pretty ad hoc and relying a lot on alpha testers to report problems.

i can see it being pretty hard to think of all the things that should be screenshot because some really random things can break. for example objects fading to the haze color when the camera looks a certain direction, corrupt geometry on character with > X bones, bad texture transform only on stage 2 or 3. i guess even these could be picked up by luck in a particular shot.

]]> By: Aras Pranckevičius http://aras-p.info/blog/2007/07/31/testing-graphics-code/#comment-1012 Aras Pranckevičius Wed, 01 Aug 2007 06:10:38 +0000 http://aras-p.info/blog/2007/07/31/testing-graphics-code/#comment-1012 Roy: the thing is - there's no single reference card. A card might not support shaders = result is different (some effects will fallback). Or shadows = result is different (no shadows). And so on. What could be possibly done is storing a reference image set per "card generation" ("this is how it should look on ps2.0 cards"), and then do a bit of fuzzy comparison. Ryan: thanks! It does not actually happen at the engine level, all that is required from the engine is the ability to do screenshots and load "game levels" (in this case it's test scenes) in succession. Roy: the thing is - there’s no single reference card. A card might not support shaders = result is different (some effects will fallback). Or shadows = result is different (no shadows). And so on. What could be possibly done is storing a reference image set per “card generation” (”this is how it should look on ps2.0 cards”), and then do a bit of fuzzy comparison.

Ryan: thanks! It does not actually happen at the engine level, all that is required from the engine is the ability to do screenshots and load “game levels” (in this case it’s test scenes) in succession.

]]> By: Ryan http://aras-p.info/blog/2007/07/31/testing-graphics-code/#comment-1007 Ryan Tue, 31 Jul 2007 23:47:08 +0000 http://aras-p.info/blog/2007/07/31/testing-graphics-code/#comment-1007 Great work! I work at a studio that used to contract for EA and was recently bought up, and I've wanted to do this for every game I've ever worked on. It's much more cost-effective to do it at an engine level as you're doing, and I'm so glad to hear that it actually works and is useful. I think I'll keep around a bookmark to this entry to show my coworkers. And FYI, EA has a compatibility testing lab for PC games that is quite rigorous. There's always a bit of dread when submitting a build for compatibilty testing, because we invariably get strange bugs on the most obscure video cards. Great work!

I work at a studio that used to contract for EA and was recently bought up, and I’ve wanted to do this for every game I’ve ever worked on. It’s much more cost-effective to do it at an engine level as you’re doing, and I’m so glad to hear that it actually works and is useful. I think I’ll keep around a bookmark to this entry to show my coworkers.

And FYI, EA has a compatibility testing lab for PC games that is quite rigorous. There’s always a bit of dread when submitting a build for compatibilty testing, because we invariably get strange bugs on the most obscure video cards.

]]> By: Roy http://aras-p.info/blog/2007/07/31/testing-graphics-code/#comment-1006 Roy Tue, 31 Jul 2007 23:24:49 +0000 http://aras-p.info/blog/2007/07/31/testing-graphics-code/#comment-1006 Don't some people also do 'fuzzy compares' with reference images? As in, you basically have one card be the reference card of which you're certain it renders properly. Subsequently, all other cards should reasonably match the render of the reference card. That would save quite a bit of screenshot flipping. Don’t some people also do ‘fuzzy compares’ with reference images? As in, you basically have one card be the reference card of which you’re certain it renders properly. Subsequently, all other cards should reasonably match the render of the reference card.

That would save quite a bit of screenshot flipping.

]]>