On 3/9/26 5:13 PM, Frank Steinmetzger wrote: > Right on the first post of this thread I thought “weren’t you a big fan of > my md5 directory hasher script”? :D > > > Am Mon, Mar 02, 2026 at 03:30:54PM -0600 schrieb Dale: > >> What I was hoping, I could tell it to generate a checksum for a entire >> directory, the way it does for a file. Then I do the same thing on my main >> version and see if my main version matches the backup version. I think that >> even if this could be done, it would take days to generate it. > > Definitely. But if you generate the checksum files once, then you have them > available and use them to detect bitrot. And reference them when copying > only individual directories. > >> I was hoping >> for something that takes less than a hour at least, maybe even 15 or 20 >> minutes. I'm getting the idea that what I'm wanting could take days or >> longer to perform. 42TBs of data is a lot to check. Some 64,000 files. > > Well as others mentioned, there’s no other way than reading them all in.
You can still try to avoid fork+exec, which is particularly slow if there's an extremely large number of *small* files. $ cd sys-apps/portage; find -print0 | bsdtar -cnf - --format=mtree --options='!all,indent,use-set,type,uid,gid,mode,sha256,link' --null --files-from - #mtree /set type=file uid=1000 gid=1000 mode=644 . mode=755 type=dir ./Manifest sha256digest=e8d539b1720acf5a75ba1ab9ea9a10f16ec480e2b2180ffdcc249430131d4acf ./metadata.xml sha256digest=0bdd464be5414059a65ce4d24ededb045fc82db3adef9ca02466ec00aa110b65 ./portage-3.0.72-r1.ebuild \ sha256digest=7f8eceaf4580339c9e432df705b9bc5f09354ec96af903c005269d75489e7701 ./portage-3.0.77-r1.ebuild \ sha256digest=30baee2cf0c2005572a2c03e44af2dc6af45aa4ae051a597ae5dab8d2a237683 ./portage-3.0.77-r3.ebuild \ sha256digest=3b7dc601cc4a4bcea91eb1e7a5db11a6891083fb15d8c9759cb073c696dafea0 ./portage-3.0.77.ebuild \ sha256digest=67bfd907dc2339205fd77c79b3a549e1c7c71aaefdbf48a4f678961edeee3bd0 ./portage-9999.ebuild \ sha256digest=59b4fb52c84f358594c8aa8b73d372d67fcbe64cbc5fa4b0807cc2b6a1d862a3 ./files mode=755 type=dir ./files/0001-bin-emerge-ebuild-don-t-define-signal-handlers-in-gl.patch \ sha256digest=7c382290a6091c61e6078744352d01e29e516bf6aed37ddc6b04a0a21bc6e24b ./files/0001-estrip-silence-noise-from-objcopy-if-build-ID-sectio.patch \ sha256digest=fc8c6c6da5a3bf95ff3498fee0f8329c9b6eb0959fe10e1f46cfa01d78f34b83 ./files/README.RESCUE \ sha256digest=a3f7a467a73c97d42c7c034d87b9480640677ab6dc1a318ec7bdd0dbdd67fab9 ./files/portage-ccache.conf \ sha256digest=5175a18d3b81bd0aa499e0da6a2f44b920cf26f650e8dadea8dc2dc6551f8a39 ./files/portage-tmpdir.conf \ sha256digest=f20b0c86299ad7f94c1516f925af0321ade5c32faae7ca998dc46ec295f5846a See also https://man.freebsd.org/cgi/man.cgi?query=mtree -- Eli Schwartz
OpenPGP_signature.asc
Description: OpenPGP digital signature
