On Sat, 2016 May 14 22:11+0200, Eduard Bloch wrote:
>
> Yes, they are gone. And the expiration algorithm should to ignore them
> if a better version is found, i.e. Packages.xz. Just like apt does.
> Such files are also stored there when the pdiff patcher recreated a
> copy using pdiff patches so the next run can take this version as base
> for patching.
>
> Do you see any problem with that files?
These "Packages" files had a parallel .xz version in the cache, and also
a .gz on the upstream server. Now that you mention it, it's possible the
uncompressed files were the result of pdiff patching... although
wouldn't there not be .head files then?
Whatever their provenance, acng should not have sent out e-mail about
them. The 404 errors were either expected (when the uncompressed file is
a creation of acng that does not exist on the server) or non-fatal
(when other forms of the same index file are present for expiration
purposes). Neither situation should require intervention from the admin.
> > Unlike the previous situation, none of these index files exist on
> > the upstream server. I only have empty files (and no .head file)
> > for each.
>
> Maybe because it's arm64 and many mirrors don't contain it? Do have
> arm64 clients?
Correct, my mirrors certainly don't have those files. There were also
empties for armhf, s390x, etc. and I don't have clients on those either.
> Please share more information: which mirror, what's else inside of
> your cache ("find /var/cache/apt-cacher-ng").
I'm just using the "primary" U.S. mirror servers, http.us.debian.org and
us.archive.ubuntu.com. There are other repositories as well, but the
problem is already observable with these.
> The new extension in 0.9.2 identifies a by-hash/... file referenced by
> InRelease file then it creates an empty file there. This is designed
> specially for the case where user installs xenial with their recent
> apt which does not bother to download Packages.xz from the original
> location so the file is not created ever (not visible to proxy) and
> eventually cache maintenance starts cleaning "unreferenced" stuff.
I do notice that [Ubuntu xenial] InRelease lists a bunch of things that
are actually not available from the "primary" mirror, like binary-
arm64/*, binary-powerpc/*, etc. Is acng assuming that a mirror server
can provide everything that is in InRelease?
> > I did recall seeing during a manual scan/expiration shortly after
> > installing 0.9.2 that acng was "touching" a lot of index files. I
> > suspect it was doing so in the sense of creating empty files rather
> > than just updating timestamps.
>
> "touching" means what touch(1) does.
Okay, so acng creates those empty files. It shouldn't complain, then, if
it can't download those files from the upstream. Just because InRelease
describes X doesn't mean that X can actually be downloaded from that
(same) server, at least in the Ubuntu case.
> > And if that weren't enough, when I went to the expiration page and
> > hit "Check all" and then "Delete selected" to clean up this mess,
> > the resulting list on the confirmation page included numerous valid
> > index files for amd64/i386---even though they did exist upstream,
> > had been
>
> Please send the resulting web page, my crystal ball is broken.
It's pretty big. My point is, "Check all" appears to select more files
than just those with a checkbox. None of the amd64/i386 index files had
trouble downloading, so none of those had checkboxes, and yet there they
are on the delete-confirmation page after "Check all."
> > downloaded correctly, and did not have a "Tag" checkbox on the
> > expiration page. It was even going to delete files like
> > ubuntu/dists/xenial/Release!
>
> Checkbox is displayed only once per file, maybe you needed to scroll
> up to find the first one?
I looked through the whole list, and didn't see a checkbox for any
amd64/i386 index file. And the Release file didn't have a checkbox,
because it was downloadable.
(Isn't the fact that acng was about to expire-delete the Release file
for the current Ubuntu release a pretty clear indicator in itself that
something's not right? Sure, maybe if all files on the upstream server
suddenly disappeared. But short of something extraordinary like that,
this should never happen.)
> > I ended up deleting these files from the cache manually, but as soon
> > as I ran another scan, there they were again.
>
> Correct. That's why I would like to see what exactly was found in
> your cache.
I'll send you some telemetry soon, probably off-list due to its bulk.
The behavior is consistently reproducible, much to my aggravation, but
at least that should make catching it easy.
