On Sat, Feb 11, 2017 at 09:58:10AM +0100, Richard Cochran wrote: > If I am not mistaken, then you can skip the cases val==2 and val==3, > because they are equivalent to val==4 and 6.
I took a stab at this, and you can see the result, below. My version has lower average error than yours in the interval 1 < ppb < 60000, and it uses only 8 64-bit divisions. Outside of that interval, your version has lower error. So, at the very least, you should introduce a threshold and use this algorithm for adjustments under 60 ppm. Better yet, find a way to use fewer divisions for adjustments greater and 60 ppm... Thanks, Richard --- #include <stdint.h> #define TEN9 (1000000000UL) unsigned int calc_min_integer(uint64_t ppb, uint64_t *M) { uint64_t err, m, min, n, N, p2, reg; min = TEN9; for (n = 4; n <= 7; n++) { m = n * TEN9; m = (m + ppb/2) / ppb; /*truncate to HW resolution*/ reg = (m + 8) / 16; m = reg * 16; p2 = (n * TEN9 + m/2) / m; err = ppb > p2 ? ppb - p2 : p2 - ppb; if (min >= err) { min = err; N = n; *M = m; } } return N; }