Comments on: A day well spent (encoding floats to RGBA)

By: Lost in the Triangles » Blog Archive » Encoding floats to RGBA, again

Lost in the Triangles » Blog Archive » Encoding floats to RGBA, again — Fri, 20 Jun 2008 15:56:03 +0000

[...] it looks like the quest for encoding floats to RGBA textures (part 1, part 2) did not end [...]

By: Lost in the Triangles » Blog Archive » Encoding floats to RGBA, redux

Lost in the Triangles » Blog Archive » Encoding floats to RGBA, redux — Fri, 29 Jun 2007 07:58:36 +0000

[...] has interesting comments in my earlier post. So I thought I’d share what I am using right now, and try to throw some more complexities in [...]

By: gleserg

gleserg — Fri, 29 Jun 2007 07:15:31 +0000

Oops, forgot about HTML.

After some thinking with Rej and Kravchenko, the final idea looks like:

Just before truncating to a screen buffer byte value 0..255, our number looks like

0.1234 5678 9ABC DEFG HIJK LMN // 23 bit mantissa, with exponent=0, numbers represent binary digits

frac(0.123456789ABCDEFGHIJKLMN) = 0.123456789ABCDEFGHIJKLMN
framebuffer will receive 12345678 + ((bit9>0)?1:0); // rounded

frac(0.123456789ABCDEFGHIJKLMN * 256) = frac(0.123456789ABCDEFGHIJKLMN << 8) = frac(12345678.9ABCDEFGHIJKLMN) = 0.9ABCDEFGHIJKLMN
framebuffer will receive 9ABCDEFG + ((bitH>0)?1:0); // rounded

frac(0.123456789ABCDEFGHIJKLMN * 65536) = frac(0.123456789ABCDEFGHIJKLMN << 16) = frac(123456789ABCDEFG.HIJKLMN) = 0.HIJKLMN0
framebuffer will receive HIJKLMN0

further multiplication with 16777216.0 is useless since IEEE mantissa has not enough precision.

bit 9 affects both first byte and second which is bad and causes error you mentioned. The same goes for bit H which affects 2nd and 3rd bytes.

By: gleserg

gleserg — Tue, 26 Jun 2007 07:43:51 +0000

oops, html ate my comment :)
out.rgba = (RGBADecode(RGBAEncode(z)) < z);
out.rgba = (RGBADecode(RGBAEncode(z)) > z);
both render black.
(1,2,4) were simply a mathematical replacement, maybe a bad one

By: gleserg

gleserg — Tue, 26 Jun 2007 07:34:59 +0000

I meant RGBADecode(RGBAEncode(z))!=z, component fraction values seem to overlap. Shouldn’t RGBAEncode contain something like:

out.w = frac(16777216.0*v);
v-=out.w/16777216.0;
out.z = frac(65536.0*v);
v-=out.z/65536.0;
out.y = frac(256.0*v);
v-=out.y/256.0;
out.x = v;

Of course it’s way slower than vectorized frac, but in this case
out.rgba = (RGBADecode(RGBAEncode(z))==z)
renders totally white and both of
out.rgba = (RGBADecode(RGBAEncode(z))z);
render totally black picture.

The only thing I didn’t check yet is how value rounding during texture read/writes will affect precision.

By: nearaz

nearaz — Mon, 25 Jun 2007 21:01:24 +0000

The thing is that first component encodes the value. Second encodes the fraction of the value that got rounded off (because of limited precision of the first component). And so on.

If you’re using 1,2,4 for encoding; that would mean it encodes for a 1-bit texture or something like that.

By: gleserg

gleserg — Mon, 25 Jun 2007 17:08:38 +0000

Quick check.
Didn’t test it in life, but mathematics seem to be incorrect.

Let’s test it with value 1/3 = 0.33333333333…

frac( (1/3)*float3(1,2,4) ) = float3(1/3,2/3,1/3)

but

dot( float3(1/3,2/3,1/3), float3(1,1/2,1/4) ) = 1/3 + 1/3 + 1/12 is far from original 1/3

I might misunderstood the idea though

By: ReJ

ReJ — Wed, 07 Mar 2007 22:19:00 +0000

Hmmm… I wonder if you would get the same quantinization by doing ‘dummy’ texture reads from the 256×256 texture filled with f(u,v,0,1)… but even if that would work, it’s of course 2 additional texture reads per pixel, which is basically a crap :(

By: NeARAZ

NeARAZ — Wed, 07 Mar 2007 20:13:00 +0000

…but it still won’t be exactly like the quantized version. For example, it looks like Radeon X1600 quantizes like this: x*255.0-0.55781; the last number is approximate. No idea why it’s a minus, because I would think it should be a plus, but experiments tell otherwise.

So my thinking is that the chances of getting quantized-float-myself and float-coming-from-8bit-texture equal are pretty slim.

By: ReJ

ReJ — Wed, 07 Mar 2007 14:36:00 +0000

I suppose you can simulate quantinization by scaling and clamping. For example something along the lines:

// t = sign(EncodeFloatRGBA(a)-rgba_encoded_b);

float INV_EPSILON = 127;// or 255?
d = EncodeFloatRGBA(a)-rgba_encoded_b;
t = (clamp(d*INV_EPSILON+0.5)-0.5)*2;

By: NeARAZ

NeARAZ — Wed, 07 Mar 2007 10:17:00 +0000

Does not quite work. I think this code depends on sign() returning zero in case arguments are equal; otherwise only the first component (dotted with 8) affects the result.

Now, one argument comes from a 8-bit texture, while the other is encoded on the fly. So basically they are never equal (at least I couldn’t make them equal). What would be needed here is some sort of QuantizeFloatAsYouWouldDoWhenWritingIntoATexture() function :)

By: ReJ

ReJ — Sun, 04 Mar 2007 20:40:00 +0000

... give a bit of a thought for a previous (offline) discussion. Corrected code:

// rgba_encoded_b = tex2d(s)
// x = (a >= rgba_encoded_b)

float4 t = sign(EncodeFloatRGBA(a) - rgba_encoded_b);
x = (dot(float4(8,4,2,1), t) >= 0);

By: ReJ

ReJ — Sun, 04 Mar 2007 08:52:00 +0000

Have you tried exactly the same code on CPU? My pure non-educated guess would be that such operations screw up precision anyhow…