On Thu, Jan 2, 2020 at 10:35 AM Chris Dagdigian <d...@sonsorol.org> wrote: > > - I noticed you did not test small file / metadata operations. My past > experience has found that this was the #1 cause of slowness in rsync and > other file transfers. iperf and IOR tests are all well and good but you > should run something like MDTEST to hammer the system on metadata and > small file handling. If you are moving lots of tiny files or hundreds of > thousands of directories etc this could be your problem > - Single stream over 10gig has never been great for me doing big data > movement. I get way more throughput by using rsync in parallel to > migrate multi-stream either from a single 10gig connected host or a > cluster of them
thanks. this is definitely not a metadata issue, i'm only moving 100-200 files in total. this is strictly a single stream transfer problem with rsync if i parallel the rsync, i can in fact increase the performance, but the process that's doing the transfer (part of a bigger system) can't handle that. it just seemed unfathomable to me that rsync can't transfer at wire past 1G speeds.. _______________________________________________ Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit https://beowulf.org/cgi-bin/mailman/listinfo/beowulf
