On 3/11/21 2:07 PM, Eric Blake wrote:
+ /*
+ * Add in a fudge-factor (2^53 when double is IEEE format) for
+ * all scales less than P (2^50), so that things like
+ * 12.345M with unit 1000 produce 12345000 instead of
+ * 12344999.
+ */
+ if (mul > 1e49) {
The comment says less than, the code says greater than.
An alternative patch might be writing (uint64_t)(fraction * mul + 0.5)
(that is, introduce the fudge factor after the multiplication instead
of before). Preferences?
I think I would prefer this, or for further rounding error reduction,
fma(fraction, mul, 0.5).
r~
