From what I'm gleaning from MM discussions, the core issue here is a
policy clash and misconfiguration between the PS7 infrastructure's
mandatory egress proxy and specific autopkgtests that are designed to
verify functionality without a proxy present in the environment. When
these tests attempt to access external resources (like archive files or
seeds), they remove the very proxy settings required by PS7, leading to
connection failures.
So:
* Autopkgtest network-ish failures on PS7 ("Could not open", "no
Packages file found", etc.) are likely to be this bug.
* Affected packages mentioned so far include livecd-rootfs, update-
manager, ubuntu-release-upgrader, ubuntu-boot-test. Notably, this isn't
really a bug in those packages but rather a conflict with the testing
environment as setup for PS7.
* Temporary workaround is retriggering tests until they execute on PS6,
which would allow justification for migration. This is a short term
hack and an extraordinarily inefficient use of resources, and thus maybe
not appropriate except in critical situations.
* Internal workaround for infra admins is to re-test with --shell-fail
and then modify the environment. This isn't effective to do as +1
maintenance.
* For +1 maintenance, may be better to simply identify and collect
affected packages (in description of this bug), rather than brute force
them.
** Description changed:
Canonical PS7 (prodstack 7) now requires[1] a proxy for any outgoing
connection, even to archive.ubuntu.com. This breaks assumptions made by
some of the update-manager tests[2], which expect a direct connection.
+
+ Affected packages (known so far):
+ * livecd-rootfs
+ * update-manager
+ * ubuntu-release-upgrader
+ * ubuntu-boot-test
Long thread on internal mattermost here[3].
For example:
1320s ERROR: testOriginMatcherSimple
(test_update_origin.TestOriginMatcher.testOriginMatcherSimple)
1320s ----------------------------------------------------------------------
1320s Traceback (most recent call last):
1320s File "/usr/lib/python3/dist-packages/apt/cache.py", line 565, in
update
1320s res = self._cache.update(fetch_progress, slist, pulse_interval)
1320s apt_pkg.Error: E:Failed to fetch
http://archive.ubuntu.com/ubuntu/dists/noble/InRelease Cannot initiate the
connection to archive.ubuntu.com:80 (2620:2d:4000:1::101). - connect (101:
Network is unreachable)
Curiously, these tests seem to have passed in non-amd64 architectures,
but closer inspection showed that these ran in good-old-PS6 (bos03),
where direct connections to archive.ubuntu.com (and
changelogs.ubuntu.com) are allowed.
This bug might be invalid for update-manager in the end, and we could
decide to allow such direct connections in PS7, but for the meantime, we
need a place to discuss, and reference, because we are about to have to
skip these tests for SRUs and migrations that this is blocking.
Keep in mind that forcing all these tests to go through a proxy will
also probably mean we will lose test coverage, since we are losing the
directly-connected test scenario.
Some troubleshooting was done already, here is some info.
a) provisioning of proxy settings for PS7 autopkgtest vms:
- - echo http_proxy=http://egress.ps7.internal:3128/ >> /etc/environment
- - echo https_proxy=http://egress.ps7.internal:3128/ >> /etc/environment
- - echo no_proxy=127.0.0.1,127.0.1.1,localhost,localdomain >>
/etc/environment
- - echo 'Acquire::http::Proxy \"http://egress.ps7.internal:3128/\";;' >
/etc/apt/apt.conf.d/99proxy
-
+ - echo http_proxy=http://egress.ps7.internal:3128/ >> /etc/environment
+ - echo https_proxy=http://egress.ps7.internal:3128/ >> /etc/environment
+ - echo no_proxy=127.0.0.1,127.0.1.1,localhost,localdomain >>
/etc/environment
+ - echo 'Acquire::http::Proxy \"http://egress.ps7.internal:3128/\";;' >
/etc/apt/apt.conf.d/99proxy
1. No clue where this decision was made
2.
https://autopkgtest.ubuntu.com/results/autopkgtest-questing/questing/amd64/u/update-manager/20250707_093622_8d0d0@/log.gz
3. https://chat.canonical.com/canonical/pl/g7hugd5p53nwmmibntd359z3gr
** Changed in: update-manager (Ubuntu)
Status: Confirmed => Triaged
** Changed in: apport (Ubuntu)
Importance: Undecided => Critical
** Changed in: livecd-rootfs (Ubuntu)
Status: New => Triaged
** Changed in: ubuntu-boot-test (Ubuntu)
Status: Confirmed => Triaged
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/2116119
Title:
autopkgtests fail on PS7 due to mandatory proxy
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/apport/+bug/2116119/+subscriptions
--
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
