Hello, On Tue, Jan 16, 2024 at 01:17:42AM -0500, Felix Miata wrote: > If rsync really is bugged, maybe a change of options would avoid the bug. Try > instead of -av, -rlptgoDAXUNH. Could it be that verbosity is the OOM > crippler, and > not necessarily from rsync itself, but possibly from the xterm in which rsync > is > running? Does your source contain any hard links? Do you use ACLs or xattrs?
I'm totally burned out on trying to get info out of Gene, but my experience with rsync is that use of some options can massively increase memory usage. The options covered by -a don't tend to do it (and I doubt -v does anything), but things like --delay-updates, --delete--before, --delete-after and --prune-empty-dirs do. This is because rsync normally incrementally finds files to transfer so it only keeps a certain number of entries in memory and can sync any number of files without blowing up RAM, but those options disable that strategy. Even so, rsync only needs about 100 bytes of RAM per file that is checked on source, and the size of the files doesn't matter. In desperate circumstances, file tree can be rsynced in multiple segments, e.g. one rsync for each subdir or whatever other split makes sense. Maybe also ulimit can be used to set an artificially low value on the memory that rsync is allowed to use. It will fail sooner, but hopefully before using all the system's RAM and swap and having the oom-killer intervene. Thanks, Andy -- https://bitfolk.com/ -- No-nonsense VPS hosting
