Hi Daniel, On Mon, Apr 28, 2025 at 05:20:31PM +0200, Daniel Baumann wrote: > On 4/28/25 14:58, Helmut Grohne wrote: > > I am sorry to tell you that the brittle /usr-move mitigations broke > > again. > > no worries, I'm honestly so lucky that you're able and willing to help, I'll > gladly wait for your patch.
Well, it was me who got you into this misery as I claimed that
duplicating diversions would allow us to move all the files from / to
/usr. How foolish of me.
Evidently, I am unable to fully grasp the complexity involved. In the
hope of not screwing up again, I started writing more extensive tests
than last time.
Testing
~~~~~~~
You find a testcase.sh and a Makefile.test attached. These are based on
my earlier work in #1059534. Compared to back then, I expanded the test
cases significantly. The original method spelled out cases as a way of
installing or removing packages before or after upgrading to trixie.
I've now taken this down to the dpkg level driving unpacks and
configurations directly in order to exercise more control. In
particular, I am also testing the wicked case of temporarily removing
zutils as part of the upgrade and in doing so, I can reproduce the
reported problem. Let me quote the test matrix from Makefile.test as
it's important that this matrix is exhaustive.
unpgzip-confgzip-unpzutils-confzutils-rmzutils
unpgzip-confgzip-unpzutils-rmzutils
unpgzip-unpzutils-confgzip-confzutils-rmzutils
unpgzip-unpzutils-confgzip-rmzutils
unpgzip-unpzutils-confzutils-confgzip-rmzutils
unpgzip-unpzutils-confzutils-rmzutils-confgzip
unpgzip-unpzutils-rmzutils-confgzip
unpzutils-unpgzip-confgzip-confzutils-rmzutils
unpzutils-unpgzip-confzutils-confgzip-rmzutils
unpzutils-unpgzip-rmzutils-confgzip
unpzutils-confzutils-unpgzip-confgzip-rmzutils
unpzutils-confzutils-rmzutils
unpzutils-rmzutils-unpgzip-confgzip
withzutils-deinstzutils-unpgzip-rmzutils-confgzip
withzutils-deinstzutils-unpgzip-rmzutils-unpzutils-confgzip-confzutils-rmzutils
withzutils-deinstzutils-unpgzip-rmzutils-unpzutils-confzutils-confgzip-rmzutils
withzutils-unpzutils-unpgzip-confgzip-confzutils-rmzutils
withzutils-unpzutils-unpgzip-confzutils-confgzip-rmzutils
withzutils-unpzutils-unpgzip-rmzutils-confgzip
withzutils-unpzutils-confzutils-unpgzip-confgzip-rmzutils
We always start with a bookworm installation. As gzip is essential,
that's always included. zutils is installed if the case starts with
"withzutils". Then the sequence of events is executed. "unp*" means
unpacking the relevant package (from trixie). Likewise "conf*" and "rm*"
refer to configuring and removing the package. The failure case involves
scheduling a package for removal and that's what "deinst*" does. If you
can think of any valid interaction that's not listed here, please tell.
For instance, we see no withzutils-unpgzip-*, because trixie's gzip
Conflicts with bookworm's zutils and therefore we have to upgrade
zutils (unpzutils), remove zutils (skip withzutils) or schedule
deinstallation of zutils (deinstzutils) before unpacking trixie's gzip.
Fixing
~~~~~~
The actual changes are significant. For one thing, I argued that zutils
should Breaks gzip to ensure ordering of postinst. Not sure how I got
there as there is no postinst. Meanwhile, that Breaks combined with
gzip's Conflicts causes apt to invoke that wicked code path of
temporarily uninstalling zutils that presently is broken. However, the
preinst argues that wrongly renaming is ok, because gzip will be
upgraded and fix that. Once dropping that Breaks, we may no longer
wrongly rename.
That leads us to preinst changes. How we have to rename depends on
whether gzip installs its tools in /bin or /usr/bin as those diversions
have differing targets beyond aliasing. One option would be to
repeatedly dpkg-query -S those files, but each such invocation takes
0.2s even on fast CPUs and a hot cache. Instead, I propose examining
the gzip version to compute the intended location.
Then, an initial installation may be an upgrade performed by temporary
removal. In other words, gzip may have diverted its tools. We need to
actively undo that diversion.
As mentioned earlier, we should not be using --rename, so I'm changing
that to --no-rename and as a result I have to do the renaming myself.
The original location simply is the result of dpkg-divert --truename as
the removal of gzip's diversion also uses --no-rename. The destination
location depends on whether gzip installed aliased or canonical (i.e.
the gzip version).
While looking at the code, I noticed that the upgrade path was
implemented with --no-rename as well, but not doing any rename while it
really should. I'm adding that missing rename, but I also figured why
that didn't cause any problems. Due to gzip's Conflicts that branch
really should be dead code.
Last but not least, I'm being more strict on certain conditions and have
preinst abort when it is faced with unexpected situations.
Retrospective
~~~~~~~~~~~~~
How did we get into this last failure? The problem existed in the
initial move as the test suite did not cover the relevant case. It was
driven by apt and at that time apt did not exercise the temporary
removal. The addition of Breaks via #1092737 caused apt to choose the
temporary removal path and that's why it is now practically broken.
I hope to find someone who can review this work before we apply it.
Helmut
testcase.sh
Description: Bourne shell script
TESTS= \
unpgzip-confgzip-unpzutils-confzutils-rmzutils \
unpgzip-confgzip-unpzutils-rmzutils \
unpgzip-unpzutils-confgzip-confzutils-rmzutils \
unpgzip-unpzutils-confgzip-rmzutils \
unpgzip-unpzutils-confzutils-confgzip-rmzutils \
unpgzip-unpzutils-confzutils-rmzutils-confgzip \
unpgzip-unpzutils-rmzutils-confgzip \
unpzutils-unpgzip-confgzip-confzutils-rmzutils \
unpzutils-unpgzip-confzutils-confgzip-rmzutils \
unpzutils-unpgzip-rmzutils-confgzip \
unpzutils-confzutils-unpgzip-confgzip-rmzutils \
unpzutils-confzutils-rmzutils \
unpzutils-rmzutils-unpgzip-confgzip \
withzutils-deinstzutils-unpgzip-rmzutils-confgzip \
withzutils-deinstzutils-unpgzip-rmzutils-unpzutils-confgzip-confzutils-rmzutils
\
withzutils-deinstzutils-unpgzip-rmzutils-unpzutils-confzutils-confgzip-rmzutils
\
withzutils-unpzutils-unpgzip-confgzip-confzutils-rmzutils \
withzutils-unpzutils-unpgzip-confzutils-confgzip-rmzutils \
withzutils-unpzutils-unpgzip-rmzutils-confgzip \
withzutils-unpzutils-confzutils-unpgzip-confgzip-rmzutils \
all: $(foreach t,$(TESTS),testout/$(t))
testout/%:
./testcase.sh "$*" >"$@" 2>&1; echo $$? >> "$@"
diff --git a/debian/control b/debian/control
index 0020d21..31a2a1a 100644
--- a/debian/control
+++ b/debian/control
@@ -18,11 +18,6 @@ Architecture: any
Depends:
${misc:Depends},
${shlibs:Depends},
-Breaks:
-# We must ensure that gzip is upgraded before zutils.postinst runs. As it is
-# essential, Breaks is sufficient here and the janitor may propose dropping
-# this relation eventually.
- gzip (<< 1.12-1.1~),
Suggests:
bzip2,
lzip,
diff --git a/debian/zutils.preinst b/debian/zutils.preinst
index 0b242e6..3512868 100755
--- a/debian/zutils.preinst
+++ b/debian/zutils.preinst
@@ -4,14 +4,45 @@ set -e
# DEP17 M18: Duplicate diversion in aliased location /bin.
+die() {
+ printf '%s, cannot proceed\n' "$*" 1>&2
+ exit 1
+}
+
case "${1}" in
install)
+ # Situations considered:
+ # * bookworm or earlier gzip is fully installed.
+ # -> No diversions, /bin/${FILE}
+ # * trixie or later gzip is unpacked.
+ # -> No diversions, /usr/bin/${FILE}
+ # * trixie or later gzip has been unpacked while bookworm's
+ # zutils was installed, but zutils has since been removed.
+ # -> /usr/bin/FILE -> /usr/bin/FILE.usr-is-merged
+ # * trixie or later gzip is fully installed.
+ # -> No diversions, /usr/bin/${FILE}
+ #
+ # We cannot run between gzip.preinst and gzip unpack.
+ GZIP_VERSION=$(dpkg-query -f '${Version}' -W gzip)
+ GZIP_PREFIX=/usr
+ dpkg --compare-versions "$GZIP_VERSION" lt 1.12-1.1~ &&
GZIP_PREFIX=
for FILE in zcat zcmp zdiff zegrep zfgrep zgrep
do
- # We may move $FILE to $FILE.gzip when we expected
$FILE.gzip.usr-is-merged here.
- # This is ok, because gzip will be upgraded and
overwrite $FILE.gzip.
- dpkg-divert --package zutils --quiet --add --rename
--divert "/usr/bin/${FILE}.gzip" "/usr/bin/${FILE}"
- dpkg-divert --package zutils --quiet --add --rename
--divert "/bin/${FILE}.gzip.usr-is-merged" "/bin/${FILE}"
+ TRUENAME=$(dpkg-divert --truename "/usr/bin/${FILE}")
+ if [ "${TRUENAME}" = "/usr/bin/${FILE}.usr-is-merged" ]
+ then
+ dpkg-divert --package zutils --quiet --remove
--no-rename --divert "${TRUENAME}" "/usr/bin/${FILE}"
+ elif [ "${TRUENAME}" != "/usr/bin/${FILE}" ]
+ then
+ die "unexpected diversion of /usr/bin/${FILE}
to ${TRUENAME}"
+ fi
+ # We cannot --rename, because it'll rename any existing
+ # file without checking whether the file is owned or
+ # not. Correctly compute the required rename depending
+ # on the gzip version.
+ dpkg-divert --package zutils --quiet --add --no-rename
--divert "/usr/bin/${FILE}.gzip" "/usr/bin/${FILE}"
+ dpkg-divert --package zutils --quiet --add --no-rename
--divert "/bin/${FILE}.gzip.usr-is-merged" "/bin/${FILE}"
+ mv "${DPKG_ROOT:-}${TRUENAME}"
"${DPKG_ROOT:-}${GZIP_PREFIX}/bin/${FILE}.gzip${GZIP_PREFIX:+.usr-is-merged}"
dpkg-divert --package zutils --quiet --add --rename
--divert /usr/share/man/man1/${FILE}.gzip.1.gz /usr/share/man/man1/${FILE}.1.gz
done
;;
@@ -23,12 +54,23 @@ case "${1}" in
if [ "${TRUENAME}" = "/usr/bin/${FILE}.usr-is-merged" ]
then
- # gzip.preinst duplicated the diversion for us
+
+ # This branch should be dead code. The
+ # diversion indicates that trixie's gzip has
+ # been unpacked, but trixie's gzip also
+ # conflicts with zutils (<< trixie), so either
+ # we're not installing or we are upgrading from
+ # trixie.
+ if ! [ -e "${DPKG_ROOT}${TRUENAME}" ]
+ then
+ die "diverted file ${TRUENAME} does not
exist"
+ fi
dpkg-divert --package zutils --quiet --remove
--no-rename --divert "/usr/bin/${FILE}.usr-is-merged" "/usr/bin/${FILE}"
dpkg-divert --package zutils --quiet --remove
--no-rename "/bin/${FILE}"
dpkg-divert --package zutils --quiet --add
--no-rename --divert "/usr/bin/${FILE}.gzip" "/usr/bin/${FILE}"
dpkg-divert --package zutils --quiet --add
--no-rename --divert "/bin/${FILE}.gzip.usr-is-merged" "/bin/${FILE}"
- elif [ "${TRUENAME}" != "/usr/bin/${FILE}.gzip" ]
+ mv "${DPKG_ROOT}${TRUENAME}"
"${DPKG_ROOT}/usr/bin/${FILE}.gzip"
+ elif [ "${TRUENAME}" = "/usr/bin/${FILE}" ]
then
dpkg-divert --package zutils --quiet --add
--no-rename --divert "/usr/bin/${FILE}.gzip" "/usr/bin/${FILE}"
@@ -43,6 +85,9 @@ case "${1}" in
mv "${DPKG_ROOT}${TRUENAME}"
"${DPKG_ROOT}/bin/${FILE}.gzip.usr-is-merged"
fi
fi
+ elif [ "${TRUENAME}" != "/usr/bin/${FILE}.gzip" ]
+ then
+ die "unexpected diversion of /usr/bin/${FILE}
to ${TRUENAME}"
fi
done
;;
