> On Sun, Feb 12, 2017 at 03:07:34PM +0000, Mintz, Yuval wrote: > > Your algorithm ignores the HW limitation. Consider (ppb == 1): > > your logic would output N == 7, *M == 7000000000, > > Which has perfect accuracy [N / *M is 1 / 10^9]. > > But the solution for > > 'period' * 16 + 8 == 7 * 10^9 > > isn't a whole number, so this result doesn't really reflect the actual > > approximation error since we couldn't configure it to HW. > > Ok, so change my code to read: > > /*truncate to HW resolution*/ > reg = (m - 8) / 16; > m = reg * 16 + 8; > > Your HW will happyly accept the value of 'reg', right? > > > The original would return val == 1, period == 62499999; While this > > does have some error [val / (period * 16 + 8) is slightly bigger than > > 1 / 10^9, error at 18[?] digit after dot], it's the best we can > > configure for the HW. > > That is *not* the best you can do:
I have already conceded that in a different reply to this thread. In light of the difference I re-checked our calculation and found that we lose precision due to the later divisions' truncation. I.e., currently: Diff == | ppb - 10^9 * val / (period * 16 +8)| Which can be transformed into: Diff == | ppb * (period * 16 + 8) - 10^9 * val | It would reduce the approximation error as well as remove 2 divisions from each iteration. I'll send v5 with it later today.
