Btw:
The /etc/rc.d/zpool script looks into these cachefiles:
for cachefile in /etc/zfs/zpool.cache /boot/zfs/zpool.cache; do
I didn’t check where the cachefile pool property is used.
Hope this helps resolving the issue. Or maybe helps you to provide more information about your setup.
Regards,
Ronald.
Van: Ronald Klop <ronald-li...@klop.ws>
Datum: 28 november 2024 14:50
Aan: Dennis Clarke <dcla...@blastwave.org>
CC: Current FreeBSD <freebsd-current@freebsd.org>
Onderwerp: Re: zpools no longer exist after boot
Are the other disks available at the moment the boot process does zpool import?
Regards,
Ronald
Van: Dennis Clarke
Datum: 28 november 2024 14:06
Aan: Current FreeBSD
Onderwerp: zpools no longer exist after boot
This is a baffling problem wherein two zpools no longer exist after
boot. This is :
titan# uname -apKU
FreeBSD titan 15.0-CURRENT FreeBSD 15.0-CURRENT #1
main-n273749-4b65481ac68a-dirty: Wed Nov 20 15:08:52 GMT 2024
root@titan:/usr/obj/usr/src/amd64.amd64/sys/GENERIC-NODEBUG amd64 amd64 1500027
1500027
titan#
titan# zpool list
NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH
ALTROOT
t0 444G 91.2G 353G - - 27% 20% 1.00x ONLINE -
titan#
The *only* zpool that seems to exist in any reliable way is the little
NVME based unit for booting. The other two zpools vanished and yet the
devices exist just fine :
titan#
titan# camcontrol devlist
at scbus0 target 0 lun 0 (pass0,ada0)
at scbus1 target 0 lun 0 (pass1,ada1)
at scbus2 target 0 lun 0 (ses0,pass2)
at scbus6 target 0 lun 0 (ses1,pass3)
at scbus7 target 0 lun 1 (pass4,nda0)
at scbus8 target 0 lun 0 (da0,pass5)
titan#
titan# nvmecontrol devlist
nvme0: SAMSUNG MZVKW512HMJP-000L7
nvme0ns1 (488386MB)
titan#
titan# zpool status t0
pool: t0
state: ONLINE
status: Some supported and requested features are not enabled on the pool.
The pool can still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
the pool may no longer be accessible by software that does not support
the features. See zpool-features(7) for details.
scan: scrub repaired 0B in 00:00:44 with 0 errors on Wed Feb 7 09:56:40 2024
config:
NAME STATE READ WRITE CKSUM
t0 ONLINE 0 0 0
nda0p3 ONLINE 0 0 0
errors: No known data errors
titan#
Initially I thought the problem was related to cachefile being empty for
these zpools. However if I set the cachefile to something reasonable
then the cachefile property vanishes at a reboot. The file, of course, exists
just fine :
titan# zpool get cachefile proteus
NAME PROPERTY VALUE SOURCE
proteus cachefile - default
titan#
titan# zpool set cachefile="/var/log/zpool_cache" proteus
titan# zpool get cachefile proteus
NAME PROPERTY VALUE SOURCE
proteus cachefile /var/log/zpool_cache local
titan# ls -ladb /var/log/zpool_cache
-rw-r--r-- 1 root wheel 1440 Nov 28 11:45 /var/log/zpool_cache
titan#
So there we have 1440 bytes of data in that file.
titan# zpool set cachefile="/var/log/zpool_cache" t0
titan# zpool get cachefile t0
NAME PROPERTY VALUE SOURCE
t0 cachefile /var/log/zpool_cache local
titan#
titan# ls -ladb /var/log/zpool_cache
-rw-r--r-- 1 root wheel 2880 Nov 28 11:46 /var/log/zpool_cache
titan#
Now we have 2 * 1440 bytes = 2880 bytes of some zpool cache data.
titan# zpool set cachefile="/var/log/zpool_cache" leaf
titan# zpool get cachefile leaf
NAME PROPERTY VALUE SOURCE
leaf cachefile /var/log/zpool_cache local
titan#
titan# zpool get cachefile t0
NAME PROPERTY VALUE SOURCE
t0 cachefile /var/log/zpool_cache local
titan#
titan# zpool get cachefile proteus
NAME PROPERTY VALUE SOURCE
proteus cachefile /var/log/zpool_cache local
titan#
titan# reboot
From here on ... the only zpool that exists after boot is the local
little NVME samsung unit.
So here I can import those pools and then see that the cachefile property has
been wiped out :
titan#
titan# zpool import proteus
titan# zpool import leaf
titan#
titan# zpool list
NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH
ALTROOT
leaf 18.2T 984K 18.2T - - 0% 0% 1.00x ONLINE -
proteus 1.98T 361G 1.63T - - 1% 17% 1.00x ONLINE -
t0 444G 91.2G 353G - - 27% 20% 1.00x ONLINE -
titan#
titan# zpool get cachefile leaf
NAME PROPERTY VALUE SOURCE
leaf cachefile - default
titan#
titan# zpool get cachefile proteus
NAME PROPERTY VALUE SOURCE
proteus cachefile - default
titan#
titan# zpool get cachefile t0
NAME PROPERTY VALUE SOURCE
t0 cachefile - default
titan#
titan# ls -l /var/log/zpool_cache
-rw-r--r-- 1 root wheel 4960 Nov 28 11:52 /var/log/zpool_cache
titan#
The cachefile exists and seems to have grown in size.
However a reboot will once again provide nothing but the t0 pool.
Baffled.
Any thoughts would be welcome.
--
--
Dennis Clarke
RISC-V/SPARC/PPC/ARM/CISC
UNIX and Linux spoken
