I'm already using rsync for very large transfers (mainatining mirrors
of the rpmfind servers over the transatlantic line) without troubles
and I was very surprized to discover that for smaller dataset (aroun 7Gigs)
but made of approximately a 500,000 small files, sometimes rsync simply
blocks.
Environemnt:
- RedHat 6.2 smp PPro200 (server) and smp PIII (client)
- rsync-2.4.1-2 i386 RPM as shipped in RedHat
- fast lan (machines not on the same subnet but both on
MIT network)
- kernel 2.2.16-17 from the beta RH-7.0 but with ext3
extension
- mirroring the content of W3C web space
- though the rsync processes did grow a lot there was still
some memory left (not trashing)
- I did grew /proc/sys/inode-max and /proc/sys/file-max to
values quite larger than the default setups (happened on
default setup too).
Symptoms:
- using rsync in client/server mode, not through ssh/rsh
- this occurs in the transfer phase after the file list exchange
finished
- this is not repoducible, i.e. on 5 attempts today, the rsync
managed to go further, it didn't block on a specific file,
but always blocked at some point, repeating the process allow
to go further.
- it usually happens when transfering a large amount of small
files (like a series of thousands of small gif representing glyphs)
- all three processes halts, they don't seem to consume CPU at all
in top, I always ended up killing the process, network works
and between both machines
Debugging:
- not much I had not a gdb ready copy when this happened
- on all three process, attaching strace showed them waiting on
select() exiting it on timeout and rentering it.
I really think it's related to the very large number of files,
I don't have troubles mirroring 50Gigs fully loaded disks but that's
the first time I run rsync on a set of file of taht magnitude.
If this is not a known problem I will try to get more debugging
informations,
yours,
Daniel
--
[EMAIL PROTECTED] | W3C, INRIA Rhone-Alpes | Today's Bookmarks :
Tel : +33 476 615 257 | 655, avenue de l'Europe | Linux XML libxml WWW
Fax : +33 476 615 207 | 38330 Montbonnot FRANCE | Gnome rpm2html rpmfind
http://www.w3.org/People/all#veillard%40w3.org | RPM badminton Kaffe
