Hi, Adding Martin and Antonio into the loop, as I don't know the details of autopkgtest so well. Question is near the bottom.
On 25-06-18 00:24, Ian Jackson wrote:
> (See also #868550 which discusses a race with a much lower occurrence
> probability and and different symptoms: the race in #868550 can be
> worked around by simply retrying the failed gnupg2 invocation, whereas
> the race I am reported in this bug apparent cannot.)
>
> Paul Gevers writes ("dgit in unstable"):
>> Just spotted the state of dgit in autopkgtest on unstable. I assume that
>> you are aware. (No regression spotted yet in testing).
>
> This is almost certianly a bug in gnupg2. It works with my up to date
> sid chroot when run with stretch's adt-run on dgit_5.1.dsc. [1]
>
> So I think it must be a race or some other stochastic bug or maybe
> some weird kernel interaction (my testbed is an schroot and I'm
> running stretch's amd64 kernel).
>
> ci.debian.net is losing this race every time in unstable but not for
> testing. None of the package version changes look relevant (see
> below).
>
> As I write in #868550 I have sometimes seen similar symptoms, which
> are generally resolved by saying `killall gpg-agent'.
>
>
> gnupg2 maintainers: What should we do to debug this ?
>
> It is very disappointing that I first reported problems of this kind
> in October 2016 in #841143, and although that was closed in January
> 2017, I found in July 2017 that there were still problems and opened
> #868550. That was a year ago. If gnupg2 cannot be made to work
> reliably, can we switch back to gnupg1 as default ?
>
>
> Paul, is it possible that the test runner somehow has a leftover
> gpg-agent ? What, if anything, is shared with previous test runs ?
If I recall a discussion well, than the testbed is only shared if the
test dependencies are the same or a super set of previous run and
"breaks-testbed" is not set. Martin, do you confirm?
> Incidentally, I don't understand why it has suddenly started failing
> much more often with so little change to any of the packages.
>
> I looked at this
> https://ci.debian.net/data/packages/unstable/amd64/d/dgit/495400.log
> which is the debci log of the first failing run in unstable and the
> updates are:
Please, don't rely on these logs (bug 812856). If you want a conclusive
answer, inspect the autopkgtest artifacts. I have put my output at the
bottom, although the below list appears to be correct.
> -linux-libc-dev 4.16.16-1
> +linux-libc-dev 4.16.16-2
> -linux-libc-dev 4.16.16-1
> +linux-libc-dev 4.16.16-2
> -binutils 2.30-21
> -binutils-common 2.30-21
> -binutils-x86-64-linux-gnu 2.30-21
> +binutils 2.30-22
> +binutils-common 2.30-22
> +binutils-x86-64-linux-gnu 2.30-22
> -libbinutils 2.30-21
> +libbinutils 2.30-22
> -libpam-systemd 238-5
> +libpam-systemd 239-1
> -libsystemd0 238-5
> +libsystemd0 239-1
> -libudev1 238-5
> +libudev1 239-1
> -systemd 238-5
> -systemd-sysv 238-5
> +systemd 239-1
> +systemd-sysv 239-1
>
> Has something else changed ?
>
> Ian.
>
> [1] My complete adt-run has not yet finished but it's obviously not
> behaving like the one on ci.debian.net.
Paul
PS: for you all to know, I triggered a test with git from experimental
(already before this bug report). It is possible now with the new debci
API. (triggers with spaces are not properly reported on the results
page, I already knew that but I'll file a bug about it nevertheless). I
still intend to automate this in the future, but if one wants...
elbrus@respighi:~$ cat tmp.json
[{"package": "dgit", "trigger": "try git from experimental",
"pin-packages":[["src:git", "experimental"]]}]
elbrus@respighi:~$ curl --cacert /etc/ssl/ca-global/ca-certificates.crt
--header "Auth-Key: $(cat debci.key)" --form tests=@tmp.json
"https://ci.debian.net/api/v1/test/unstable/amd64";
==== artifacts diff =====
The first number are the duration and exit code. (I note the order of
the kernel report changed, hence a spurious comma in two lines).
paul@testavoira /tmp $ diff -r dgit_pass dgit_fail | grep -e '<' -e '>'
| sort --unique
< 1455
< 2
> 2927
> 6
> badcommit-rewrite FAIL non-zero exit status 16
< badcommit-rewrite PASS
< binutils 2.30-21
> binutils 2.30-22
< binutils-common 2.30-21
> binutils-common 2.30-22
< binutils-x86-64-linux-gnu 2.30-21
> binutils-x86-64-linux-gnu 2.30-22
> clone-nogit FAIL non-zero exit status 16
< clone-nogit PASS
< "cpu_flags": "fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge
mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx rdtscp lm
constant_tsc rep_good nopl xtopology eagerfpu pni pclmulqdq ssse3 cx16
pcid sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c
rdrand hypervisor lahf_lm kaiser fsgsbase smep erms xsaveopt"
> "cpu_flags": "fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge
mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx rdtscp lm
constant_tsc rep_good nopl xtopology eagerfpu pni pclmulqdq ssse3 cx16
pcid sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c
rdrand hypervisor lahf_lm kaiser fsgsbase smep erms xsaveopt",
> debpolicy-dbretry FAIL non-zero exit status 16
< debpolicy-dbretry PASS
> debpolicy-newreject FAIL non-zero exit status 16
< debpolicy-newreject PASS
> debpolicy-quilt-gbp FAIL non-zero exit status 16
< debpolicy-quilt-gbp PASS
> defdistro-rpush FAIL non-zero exit status 16
< defdistro-rpush PASS
> distropatches-reject FAIL non-zero exit status 16
< distropatches-reject PASS
> downstream-gitless FAIL non-zero exit status 16
< downstream-gitless PASS
> dpkgsourceignores-correct FAIL non-zero exit status 16
< dpkgsourceignores-correct PASS
> drs-clone-nogit FAIL non-zero exit status 16
< drs-clone-nogit PASS
> drs-push-masterupdate FAIL non-zero exit status 16
< drs-push-masterupdate PASS
> drs-push-rejects FAIL non-zero exit status 16
< drs-push-rejects PASS
> dsd-clone-nogit FAIL non-zero exit status 16
< dsd-clone-nogit PASS
> gbp-orig FAIL non-zero exit status 16
< gbp-orig PASS
> gdr-diverge-nmu-dgit FAIL non-zero exit status 16
< gdr-diverge-nmu-dgit PASS
> gdr-edits FAIL non-zero exit status 16
< gdr-edits PASS
> gdr-import-dgit FAIL non-zero exit status 16
< gdr-import-dgit PASS
> gdr-subcommands FAIL non-zero exit status 16
< gdr-subcommands PASS
> gitworktree FAIL non-zero exit status 16
< gitworktree PASS
> import-dsc FAIL non-zero exit status 16
< import-dsc PASS
> import-native FAIL non-zero exit status 16
< import-native PASS
> inarchivecopy FAIL non-zero exit status 16
< inarchivecopy PASS
< "kernel_version": "Linux 4.9.0-5-amd64 #1 SMP Debian 4.9.65-3+deb9u2
(2018-01-04)",
> "kernel_version": "Linux 4.9.0-6-amd64 #1 SMP Debian 4.9.82-1+deb9u3
(2018-03-02)",
< libbinutils 2.30-21
> libbinutils 2.30-22
< libpam-systemd 238-5
> libpam-systemd 239-1
< libsystemd0 238-5
> libsystemd0 239-1
< libudev1 238-5
> libudev1 239-1
< linux-libc-dev 4.16.16-1
> linux-libc-dev 4.16.16-2
> mirror-debnewgit FAIL non-zero exit status 16
< mirror-debnewgit PASS
> mirror FAIL non-zero exit status 16
< mirror PASS
> mirror-private FAIL non-zero exit status 16
< mirror-private PASS
> mismatches-dscchanges FAIL non-zero exit status 16
< mismatches-dscchanges PASS
> multisuite FAIL non-zero exit status 16
< multisuite PASS
> newtag-clone-nogit FAIL non-zero exit status 16
< newtag-clone-nogit PASS
< "nproc": "2",
> "nproc": "2"
> oldnewtagalt FAIL non-zero exit status 16
< oldnewtagalt PASS
> oldtag-clone-nogit FAIL non-zero exit status 16
< oldtag-clone-nogit PASS
> orig-include-exclude-chkquery FAIL non-zero exit status 16
< orig-include-exclude-chkquery PASS
> orig-include-exclude FAIL non-zero exit status 16
< orig-include-exclude PASS
> overwrite-chkclog FAIL non-zero exit status 16
< overwrite-chkclog PASS
> overwrite-junk FAIL non-zero exit status 16
< overwrite-junk PASS
> overwrite-splitbrains FAIL non-zero exit status 16
< overwrite-splitbrains PASS
> overwrite-version FAIL non-zero exit status 16
< overwrite-version PASS
> protocol-compat FAIL non-zero exit status 16
< protocol-compat PASS
> push-buildproductsdir FAIL non-zero exit status 16
< push-buildproductsdir PASS
> push-newpackage FAIL non-zero exit status 16
< push-newpackage PASS
> push-nextdgit FAIL non-zero exit status 16
< push-nextdgit PASS
> push-source FAIL non-zero exit status 16
< push-source PASS
> push-source-with-changes FAIL non-zero exit status 16
< push-source-with-changes PASS
> quilt FAIL non-zero exit status 16
> quilt-gbp FAIL non-zero exit status 16
< quilt-gbp PASS
< quilt PASS
> quilt-singlepatch FAIL non-zero exit status 16
< quilt-singlepatch PASS
> quilt-splitbrains FAIL non-zero exit status 16
< quilt-splitbrains PASS
> rpush FAIL non-zero exit status 16
< rpush PASS
< systemd 238-5
> systemd 239-1
< systemd-sysv 238-5
> systemd-sysv 239-1
> tag-updates FAIL non-zero exit status 16
< tag-updates PASS
> trustingpolicy-replay FAIL non-zero exit status 16
< trustingpolicy-replay PASS
> version-opt FAIL non-zero exit status 16
< version-opt PASS
< "virt_server": "autopkgtest-virt-lxc --sudo --name ci-1529681964
autopkgtest-unstable-amd64",
> "virt_server": "autopkgtest-virt-lxc --sudo --name ci-1529712769
autopkgtest-unstable-amd64",
signature.asc
Description: OpenPGP digital signature
