On 3/2/26 3:52 PM, Nate Eldredge wrote:
On Mar 2, 2026, at 14:32, Dale <[email protected]> wrote:
On 3/2/26 8:17 AM, Grant Edwards wrote:
On 2026-03-02, Dale <[email protected]> wrote:
I was hoping I could do a whole directory not each individual file.
Do you only want to know if the _directory_ has changed and not if the
contents of any file has changed? That's possible, but it might
require a small C program: I can't think of existing utility that does
that.
What I was hoping, I could tell it to generate a checksum for a entire
directory, the way it does for a file. Then I do the same thing on my main
version and see if my main version matches the backup version. I think that
even if this could be done, it would take days to generate it. I was hoping
for something that takes less than a hour at least, maybe even 15 or 20
minutes. I'm getting the idea that what I'm wanting could take days or longer
to perform. 42TBs of data is a lot to check. Some 64,000 files.
Well, inherently, in order to checksum all the data, you have to read all the
data - every byte of every file. Putting all the files in a directory doesn't
change that.
If your CPU is reasonable, this will likely be disk bound. Are your disks fast
enough to read 42 TB in 15 minutes?
That is what occurred to me as I was reading replies. In a directory or
not, it still has to read each file. It's going to take a while whether
it is done on a whole directory level or each file. There is no
reasonably fast way to do this on a directory this large.
I remember the first time I ran Franks checksum script. It took like
two weeks to generate all those. That's on my new rig. My NAS box rig
is not near as fast. Actually, it's three rigs old. My main rig is the
fastest, backup rig is the next fastest and newest, still about 10 or
more years old tho, and the NAS box is the rig before that. I think it
has 6 cores now but not super fast. It only has 12GBs of memory since
one memory stick died. I need to sent it in for replacement.
Yea, there's no fast way to do this. Well, I could build a system with
a threadripper CPU and SAS or SSD drives. If only I were that rich. ROFL
I am going to look into rsync more. I'd like it to at least checksum
new or changed files that it transfers over. If it can do that. Even
that may take to long tho.
Dale
:-) :-)
