On 2023-12-15 11:40:21, Predrag Zečević wrote:
Strange is this:
:; pkg list | grep golang
developer/golang 1.21.1-2023.0.0.0 i--
developer/golang-121 1.21.5-2023.0.0.0 i--
But:
:; ls -dlh /var/pkg/publisher/openindiana.org/pkg/developer%2Fgolang*
drwxr-xr-x 2 root root 9 Dec 13 19:36
/var/pkg/publisher/openindiana.org/pkg/developer%2Fgolang
drwxr-xr-x 2 root root 4 Aug 23 2021
/var/pkg/publisher/openindiana.org/pkg/developer%2Fgolang-115
drwxr-xr-x 2 root root 4 Aug 25 2021
/var/pkg/publisher/openindiana.org/pkg/developer%2Fgolang-116
drwxr-xr-x 2 root root 3 Jun 19 15:21
/var/pkg/publisher/openindiana.org/pkg/developer%2Fgolang-119
drwxr-xr-x 2 root root 4 Oct 4 11:24
/var/pkg/publisher/openindiana.org/pkg/developer%2Fgolang-120
drwxr-xr-x 2 root root 5 Dec 14 12:47
/var/pkg/publisher/openindiana.org/pkg/developer%2Fgolang-121
So, no directory was removed when package is removed! That explains
15 GB in /var/pkg/publisher/ directory.
If no one else have this problem, then it is not bug, but something else
Regards
Yes, this is me (somehow):
I was executing "pfexec pkg update -vn" every morning to get latest
packages available (output was send to file, parsed for interesting
packages, and sent to me per e-mail).
Seems that command downloads packages?
To prove, I did test:
:; cd /var/pkg/publisher/openindiana.org/pkg
:; find . -type f | sort > /tmp/pkg-1.txt
:; pfexec pkg update -vn # executed at 2023-12-15 10:32:47 CET
:; find . -type f | sort > /tmp/pkg-2.txt
:; diff /tmp/pkg-2.txt /tmp/pkg-1.txt | wc -l
770
So, I got 770 new entries under /var/pkg/publisher/openindiana.org/pkg
!!!
Example:
:; ls -hld ./system%2Fsecurity%2Fkerberos-5/* | wc -l
891
:; ls -hld ./system%2Fsecurity%2Fkerberos-5/* | tail -5
-rw-r--r-- 1 root root 46K Dec 11 05:00
./system%2Fsecurity%2Fkerberos-5/0.5.11%2C5.11-2023.0.0.21923%3A20231211T012803Z
-rw-r--r-- 1 root root 46K Dec 12 05:00
./system%2Fsecurity%2Fkerberos-5/0.5.11%2C5.11-2023.0.0.21925%3A20231212T013240Z
-rw-r--r-- 1 root root 46K Dec 13 19:35
./system%2Fsecurity%2Fkerberos-5/0.5.11%2C5.11-2023.0.0.21925%3A20231213T011458Z
-rw-r--r-- 1 root root 46K Dec 14 05:00
./system%2Fsecurity%2Fkerberos-5/0.5.11%2C5.11-2023.0.0.21926%3A20231214T011328Z
-rw-r--r-- 1 root root 46K Dec 15 10:33
./system%2Fsecurity%2Fkerberos-5/0.5.11%2C5.11-2023.0.0.21928%3A20231215T071813Z
:; du -sh ./system%2Fsecurity%2Fkerberos-5/
41M ./system%2Fsecurity%2Fkerberos-5/
And so on...
Am I right when assuming that 'pkg update -vn' should NOT _download_
packages?
From 'man pkg':
---<snip>-------
-n
Perform a trial run of the operation with no package
changes made.
---<snip>-------
If answer is yes, then I will open ticket...
Also I assume that I am allowed to remove any package manifest file
under /var/pkg/publisher/openindiana.org/pkg what is not installed?
Regards.
P.S: IMHO, seems *pkg* keeps installed package "registry" in this
structure, under /var/pkg/publisher/openindiana.org:
:; ls -hl catalog/
total 2,0K
lrwxrwxrwx 1 root root 65 Dec 15 10:33 catalog.attrs ->
../origins/12dad71d06df371e72f75c488f47c75a13e340d1/catalog.attrs
lrwxrwxrwx 1 root root 66 Dec 15 10:33 catalog.base.C ->
../origins/12dad71d06df371e72f75c488f47c75a13e340d1/catalog.base.C
lrwxrwxrwx 1 root root 72 Dec 15 10:33 catalog.dependency.C ->
../origins/12dad71d06df371e72f75c488f47c75a13e340d1/catalog.dependency.C
lrwxrwxrwx 1 root root 69 Dec 15 10:33 catalog.summary.C ->
../origins/12dad71d06df371e72f75c488f47c75a13e340d1/catalog.summary.C
:; ls -hl origins/12dad71d06df371e72f75c488f47c75a13e340d1/
total 346M
-rw-r--r-- 1 root root 40K Dec 15 09:16 catalog.attrs
-rw-r--r-- 1 root root 7,5M Dec 15 08:29 catalog.base.C
-rw-r--r-- 1 root root 310M Dec 15 08:29 catalog.dependency.C
-rw-r--r-- 1 root root 29M Dec 15 08:29 catalog.summary.C
So, let me update here,
I have found hundred of thousands of "orphaned" manifest files (see
message above for examples, e.g. kerberos-5 package example).
Those files have occupied 15 GB of disk space (also in each BE created
during pkg update).
Space was eaten by 'pkg update -nv' command (package not installed but
manifest is downloaded and kept) which I was running last couple years,
once/day.
This is script (if someone else needs it), which might run for long time
(like it was on my installation: more than 48 hours) and at the end it
compiles list of commands to remove orphaned manifest files:
---8<---------
#!/bin/bash
# Clean-up /var/pkg/publisher/openindiana.org/pkg directory
TODAY=$(date '+%Y%m%dT%H%M%S')
PKGMFF="/var/tmp/PKGs.${TODAY}-manifests.txt"
PKGINP="/var/tmp/PKGs.${TODAY}-installed.txt"
PKGMRM="/var/tmp/PKGs.${TODAY}-manifests-to-remove.sh"
echo "Collecting manifest data from
/var/pkg/publisher/openindiana.org/pkg/ ..."
(
cd /var/pkg/publisher/openindiana.org/pkg/
find . -type f | cut -d "/" -f 2- | sort > ${PKGMFF}
)
echo "Collecting installed packages ..."
(
pkg info | grep FMRI: | cut -d "/" -f 4- | sort > ${PKGINP}
)
# Each row:
audio%2Faudio-utilities/0.5.11%2C5.11-2020.0.0.19486%3A20200221T011730Z
# < pkg_name > < datets >
# so, loop through manifest list, get pkg_name (replace %2F with /), get
datets value
# and try to find that pkg FMRI contains datets). In case of no
result, that manifest file may be removed
(
cat ${PKGMFF} | while read MANIFEST; do
PKG_URLN=${MANIFEST%/*} # package name, but needs fix (slashes)
PKG_NAME=${PKG_URLN//%2F/\/} # replace string with slash character
PKG_DATE=${MANIFEST: -16} # TS is last 16 characters (always)
# If pkg for this manifest file is not found in system, remove it:
PKG_INST=$(pkg info pkg://openindiana.org/${PKG_NAME} |& grep
${PKG_DATE})
echo "# Result of 'pkg info pkg://openindiana.org/${PKG_NAME} |&
grep ${PKG_DATE}': '${PKG_INST}'"
if [[ -z ${PKG_INST} ]]; then
echo "pfexec rm -rf
/var/pkg/publisher/openindiana.org/pkg/${MANIFEST}"
fi
done
echo
echo "pfexec pkg refresh --full"
) |& tee ${PKGMRM}
echo "Create backup BE and execute: 'bash ${PKGMRM}'"
---8<---------
Before running commands from script, backup BE HAS to be created, in
case you want roll-back.
Luckily for me, script (and set of commands did a job properly: I was
able to update OI and reboot it this morning w/o issues).
Right now:
:; du -sh /var/pkg/publisher/openindiana.org/*
3,5K /var/pkg/publisher/openindiana.org/catalog
1,5K /var/pkg/publisher/openindiana.org/certs
1,0K /var/pkg/publisher/openindiana.org/last_refreshed
366M /var/pkg/publisher/openindiana.org/origins
280M /var/pkg/publisher/openindiana.org/pkg
Which is acceptable (I have removed backup BE, but still have one
snapshot of old BE which will go away on next update)
Hope these finding can help someone.
Next week, at some quiet time, I will check my theory, and if I am right
will open ticket...
With best regards.
_______________________________________________
openindiana-discuss mailing list
[email protected]
https://openindiana.org/mailman/listinfo/openindiana-discuss
--
Predrag Zečević
[email protected]
_______________________________________________
openindiana-discuss mailing list
[email protected]
https://openindiana.org/mailman/listinfo/openindiana-discuss
