STINNER Victor added the comment:
Not only I'm too lazy to compute manually the number of loops and repeat, but
also I don't trust myself. It's even worse when someone publishs results of a
micro-benchmark. I don't trust how the benchmark was calibrated. In my
experience, micro-benchmark are polluted by noise in timings, so results are
not reliable.
benchmarks.py calibration is based on time, whereas timeit uses hardcoded
constants (loops=1000000, repeat=3) which can be modified on the command line.
benchmarks.py has 3 main parameters:
- minimum duration of a single run (--min-time): 100 ms by default
- maximum total duration of the benchmark: benchmark.py does its best to
respect this duration, but it can be longer: 1 second by default
- minimum repeat: 5 by default
The minimum duration is increased if the clock resolution is bad (1 ms or
more). It's the case on Windows for time.clock() on Python 2 for example.
Extract of benchmark.py:
min_time = max(self.config.min_time, timer_precision * 100)
The estimation of the number of loops is not reliable, but it's written to be
"fast". Since I run a micro-benchmark many times, I don't want to wait too
long. It's not a power of 10, but an arbitrary integer number. Usually, when
running benchmark.py multiple times, the number of loops is different each
time. It's not really a big issue, but it probably makes results more difficult
to compare.
My constrain is max_time. The tested function may not have a linear duration
(time = time_one_iteration * loops).
https://bitbucket.org/haypo/misc/src/348bfd6108e9985b3c2298d2745eb5ddfe7042e6/python/benchmark.py?at=default#cl-416
Repeat a test at least 5 times is a compromise between the stability of the
result and the total duration of the benchmark.
Feel free to reuse my code to enhance time.py.
----------
_______________________________________
Python tracker <[email protected]>
<http://bugs.python.org/issue23693>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
