Fixed in DC++ 0.870.
** Changed in: dcplusplus
Status: Fix Committed => Fix Released
--
You received this bug notification because you are a member of
Dcplusplus-team, which is subscribed to DC++.
https://bugs.launchpad.net/bugs/1909861
Title:
FileReader is not thread safe on Linux
St
Looks like the benchmarks that you linked use 4-16 KB block sizes for
reading, which is far from optimal when doing regular sequential file
reads for larger files.
"Do you have any idea why in your Ubuntu 20.04 setup, 4MB buffers
performed notably worse than 1MB buffers?"
I'm not really sure, but
https://sasha-f.medium.com/why-mmap-is-faster-than-system-calls-
24718e75ab37 profiles mmap()-based file reading versus what looks like
what DC++ calls readCached().
mmap() is significantly faster in those experiments, so while my
response here was to just remove the associated code, given DC++'s
https://sourceforge.net/p/dcplusplus/code/ci/a484cbace55f671548b370135ac9fa42d5ddfc02/
implements this. As discussed in this bug/issue, it's mostly a no-op for
supported Win32-based DC++ platforms, but hopefully the buffer size
increase helps hashing speeds in general in a way consistent with your
I ran some additional tests with different read buffer sizes on
different platforms. Cached hashing speed on macOS is CPU-bound and the
same most likely applies to Windows results as well (both disks are able
to read 1000 MB/s+). 1024 KB buffer size looks quite good based on the
tests.
macOS 11 (
Apropos "So possibly the FileReader methods should rather be called
readAsync (overlapped/AIO) and readUnbuffered
(FILE_FLAG_NO_BUFFERING/O_DIRECT/F_NOCACHE). Is buffered reading even
needed there? When would that work if the unbuffered read fails?":
An unbuffered read might depend on buffer align
It appears that pre-uring AIO is being supplanted by io_uring
(https://unixism.net/2020/04/io-uring-by-example-part-1-introduction/
and https://www.scylladb.com/2020/05/05/how-io_uring-and-ebpf-will-
revolutionize-programming-in-linux/). AIO has had persistent critiques
since it was introduced (e.g
FileReader::readDirect implementation with O_DIRECT/F_NOCACHE:
https://github.com/airdcpp/airdcpp-
windows/commit/eb1e075d5a69862c4f2255f9ea2f204bb70921ba
However, I don't think that the main point of the low-level
FileReader::readDirect implementation for Windows is skipping of the OS
buffers (co
AirDC++ actually has some kind of support for opening files O_DIRECT:
https://github.com/airdcpp/airdcpp-
windows/blob/b863d8626d95d0ee483572a5139f8f569b558c3f/airdcpp/airdcpp/File.cpp#L380-L394
(BUFFER_NONE isn't currently being used anywhere though when opening
files)
FileReader::readCached curr
As far as
"I'm not sure if mapped reading even makes sense in cases where the file
is only being read sequentially through once (maybe things were
different back in the days when the code was written?)."
One possible explanation is that all of those mechanisms appeared no
earlier than 2000 or 200
This patch is relatively aggressive in removing FileReader::readCached()
and all associated infrastructure entirely. I'm not yet certain that's
the best, but in the absence of enough ongoing development to support
something evidently flawed.
Since non-Windows doesn't support readDirect() to avoid
Yeah this crash only happened for people using the Linux version of
AirDC++. FileReader::readMapped has always been disabled on Windows.
I'm not sure if mapped reading even makes sense in cases where the file
is only being read sequentially through once (maybe things were
different back in the day
It looks in DC++ on Windows, the only supported DC++ platform,
FileReader::readMapped() already always fell through to
FileReader::readCached():
ret = readMapped(file, callback);
if(ret == READ_FAILED) {
dcdebug("Reading [full] %s\n", file.c
13 matches
Mail list logo
