I ran some additional tests with different read buffer sizes on different platforms. Cached hashing speed on macOS is CPU-bound and the same most likely applies to Windows results as well (both disks are able to read 1000 MB/s+). 1024 KB buffer size looks quite good based on the tests.
macOS 11 (MacBook Pro, SSD) 256 KB: Cached: 2 files (20.76 GiB) have been hashed in 59 seconds (357.97 MiB/s) F_NOCACHE: 2 files (20.76 GiB) have been hashed in 1 minute 54 seconds (186.08 MiB/s) 1024 KB: Cached: 2 files (20.76 GiB) have been hashed in 1 minute 0 second (354.01 MiB/s) F_NOCACHE: 2 files (20.76 GiB) have been hashed in 1 minute 17 seconds (275.21 MiB/s) 4096 KB: Cached: 2 files (20.76 GiB) have been hashed in 59 seconds (355.90 MiB/s) F_NOCACHE: 2 files (20.76 GiB) have been hashed in 1 minute 6 seconds (320.85 MiB/s) Ubuntu 20.04 (Celeron N3150, internal HDD) 256 KB: Cached: 671 files (21.70 GiB) in 9 directories have been hashed in 3 minutes 58 seconds (93.08 MiB/s) O_DIRECT: 671 files (21.70 GiB) in 9 directories have been hashed in 4 minutes 10 seconds (88.62 MiB/s) 1024 KB: POSIX_FADV_DONTNEED: 671 files (21.70 GiB) in 9 directories have been hashed in 3 minutes 55 seconds (94.55 MiB/s) O_DIRECT: 671 files (21.70 GiB) in 9 directories have been hashed in 4 minutes 1 second (91.95 MiB/s) 4096 KB: POSIX_FADV_DONTNEED: Hashing finished: 671 files (21.70 GiB) in 9 directories have been hashed in 4 minutes 58 seconds (74.42 MiB/s) O_DIRECT: 671 files (21.70 GiB) in 9 directories have been hashed in 4 minutes 55 seconds (75.24 MiB/s) Windows 10 (AMD 2700X, SSD) OVERLAPPED, 256 KB: 191 files (21,36 GiB) have been hashed in 39 seconds (551,41 MiB/s) OVERLAPPED, 1024 KB: 191 files (21,36 GiB) have been hashed in 37 seconds (577,62 MiB/s) OVERLAPPED, 4096 KB: 191 files (21,36 GiB) have been hashed in 37 seconds (581,33 MiB/s) -- You received this bug notification because you are a member of Dcplusplus-team, which is subscribed to DC++. https://bugs.launchpad.net/bugs/1909861 Title: FileReader is not thread safe on Linux Status in DC++: New Bug description: FileReader::readMapped currently modifies the global SIGBUS handler in order to catch read errors: https://sourceforge.net/p/dcplusplus/code/ci/default/tree/dcpp/FileReader.cpp#l289 Since the function can be called concurrently from different threads (currently hashing/queue recheck/sfv check in DC++) and each of them sets and resets the SIGBUS handler, there's a high risk that the application will crash in case of read errors as they aren't being handler properly. More information about the caveats: https://www.sublimetext.com/blog/articles/use-mmap-with-care These issues are much more likely to happen with AirDC++ as it uses multiple threads for hashing. Read errors caused rather nasty crashes with corrupted stack traces for one user, but luckily he was able to catch the SIGBUS signal with gdb. I didn't even spend time in trying to figure out how to make the mapped reads work properly, as based on my testing the basic FileReader::readCached function is noticeably faster: readMapped: 671 files (21.70 GiB) in 9 directories have been hashed in 4 minutes 21 seconds (84.87 MiB/s) readCached: 671 files (21.70 GiB) in 9 directories have been hashed in 3 minutes 58 seconds (93.08 MiB/s) FileReader::readMapped is now disabled in AirDC++, as I can't see any benefits from using it. The included setjmp.h header is even causing issues when using clang for compiling on Linux: https://bugs.gentoo.org/731676 To manage notifications about this bug go to: https://bugs.launchpad.net/dcplusplus/+bug/1909861/+subscriptions _______________________________________________ Mailing list: https://launchpad.net/~linuxdcpp-team Post to : linuxdcpp-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~linuxdcpp-team More help : https://help.launchpad.net/ListHelp
