Hi,
I was working on giving at least committers, and perhaps a few more folks,
access to our custom compute setup for running CI, as I had gotten a few
reports about folks running out of CI cycles during busy periods like right
now.
I was thinking that we had enough compute credits to make that possible. Which
I luckily, prodded by a question from Melanie, did check.
Unfortunately it seems that our usage has increased substantially enough to
make that a problem, even with just cfbot.
Most of the test are quite cheap to run, however windows costs a fair bit in
licensing cost (we spend 2x on windows than we do for compute, ram and storage
for all the tests together). There's a few other things we can improve
(e.g. we spend a fair bit on cross region data transfers, which should largely
be fixable). But we do need to get the increases in windows runtime under
control.
Here's a look at the test timings. Unfortunately the trends are decidedly not
good, particularly on windows. Looking at a few random cross-branch commits
and looking only at the test times (since build times are affected by caching
etc).
CI msvc CI mingw CI linux andres-local
17 11:21 10:38 3:32 1:22
18 16:03 14:22 5:39 1:38
master 18:13 15:00 6:69 1:49
I checked this across a few commits and got similar numbers
everywhere. There's a lot of noise, but the trendline is pretty darn clear.
Due to the number of times we run the main regression tests, they have a
particularly large effect on test resources.
time cycles syscalls
17 37.13 239.2T 1.573M
18 44.27 295.7T 1.715M
master 48.19 323.5T 1.854M
Another interesting aspect is the time spent doing initdb has gone up
noticeably:
assert optimize
17 1.934 819.1
18 2.190 827.9
master 2.274 814.5
Which, given the number of times initdb is executed in the tests (even with
the template mechanism in place), probably is a factor in all this. It's
certainly interesting that somehow assertions play a major role, given that
the non-assert time hasn't regressed.
Some increase in test cycles and time is inevitable, but to me the current
rate of increase is not sustainable / faster than the rate at which hardware
speed increases.
Greetings,
Andres Freund
