Søren Sandmann <[email protected]> writes: > If we add the additional assumption that the intervals between > corresponding to successive integers must be a constant length, then it > follows from f(0x0000) = 0.0 and f(0xffff) = 1.0 that f(i) = i / > 65535.0. The easiest way to see that is to consider a conversion to > 2-bit integers instead: > > 0.0 1.0 > |-----------------------------------------------| > 0b00 0b01 0b10 0b11
The picture I drew on my whiteboard yesterday was similarly for a 2-bit
integer, but I drew bars for the ranges of floating-point numbers that
would map to each integer. Like so:
+-------+---------------+---------------+-------+
0b00 | 0b01 | 0b10 | 0b11
+-------+---------------+---------------+-------+
Then, my drawing for cairo's current algorithm was:
+-----------+-----------+-----------+-----------+
0b00 | 0b01 | 0b10 | 0b11
+-----------+-----------+-----------+-----------+
Note that positions for each integer are the same with both approaches.
Then, yes, what you are saying is true: If we wanted to reduce cairo's
error for the roundtrip conversion starting with a floating-point value,
and we wanted to do this by only changing the integer-to-floating-point
mapping, then we would end up with (as you drew):
> 0.0 1.0
> |-----------------------------------------------|
> 0b00 0b01 0b10 0b11
and with my bars:
+-----------+-----------+-----------+-----------+
| 0b00 | 0b01 | 0b10 | 0b11 |
+-----------+-----------+-----------+-----------+
And yes, that would be undesirable in that we would lose access to the
floating-point extremes when converting from integers.
Color me convinced. Your change minimizes the error without introducing
unwanted deviations in the floating-point values obtained from integers.
> The main motivation is that I want to extend pixman's test suite with
> more tolerance based tests, where a reference pixel is computed in
> floating point and then compared to pixman's output. It's highly
> desirable here that the _reference_ computation doesn't introduce its
> own errors.
Thanks for explaining. I'm in favor of the change, (and particularly
since the change will be backed by additional testing).
-Carl
--
[email protected]
pgp1dnDTTP9Ey.pgp
Description: PGP signature
_______________________________________________ Pixman mailing list [email protected] http://lists.freedesktop.org/mailman/listinfo/pixman
