Corinna Vinschen via Cygwin-apps wrote:
On May 12 16:14, Jon Turney wrote:
On 08/05/2021 21:03, Christian Franke wrote:
[...]
+bool io_stream_cygfile::compact_os_is_available = (OSMajorVersion () >= 10);
The documentation seems a bit vague, but are we really expecting this to
work on Windows 10 1507?
I think this could even work under 8.1 from what I can see on MSDN.
I skipped all Win8*, so I didn't test with 8.1 :-)
This page says "Available starting with Windows 10":
https://docs.microsoft.com/en-us/windows-hardware/drivers/ddi/ntifs/ns-ntifs-_file_provider_external_info_v0
It also says "Header: ntifs.h" but in recent "Windows Kits" all required
defines are in winioctl.h.
These defines are enabled even for '>= _WIN32_WINNT_WIN7'. According to
a test I did some time ago, Win7 could not read these files.
+{
+ const char * const p = name.c_str();
+ if (!(!strncmp (p, "/bin/", 5) || !strncmp (p, "/sbin/", 6) || !strncmp (p,
"/usr/", 5)))
+ return true; /* File is not in R/O tree. */
+ const size_t len = name.size(); /* >= 5 */
+ if (!strcmp (p + (len - 4), ".dll") || !strcmp (p + (len - 3), ".so"))
+ return true; /* Rebase will open file for writing which uncompresses the
file. */
+ if (!strcmp (p + (len - 3), ".gz") || !strcmp (p + (len - 3), ".xz"))
+ return true; /* File is already compressed. */
Is this an assertion that there are no .bz2, .lzma, .zst etc. files in the
install?
Another question is this: FILE_PROVIDER_COMPRESSION_LZX
"This algorithm is designed to be highly compact, and provides for small
footprint for infrequently accessed data."
When running a shell script, certain executables (especially coreutils,
gawk, sed, grep, find) are not so very infrequently accessed. Is this
compression really feasible for these binaries? Did you compare shell
script performance with non-compressed, XPRESS16K and LZX compressed
/bin dir?
Good point. Now I did a test with a ./configure script run after reboot:
There was significant difference with /bin/*.exe (only) uncompressed,
NTFS-, XPRESS16K- or LZX-compressed. Time was always around 23s.
Here a read speed test with fast and slow storage and a 10+ years old
i7-2600K (4C/8T). The 256MiB test file was generated by concatenating
various EXE files. All file accesses were the first after reboot. AV
(defender) was turned off:
Compression MiB T1 T2 T3,T4
======================================
None 256 0.69s 10.1s <0.02s
NTFS 159 1.03s 8.1s <0.02s
XPRESS4K 138 -
XPRESS8K 128 -
XPRESS16K 123 0.64s 5.4s <0.02s
LZX 97 0.79s 4.8s <0.02s
T1,T2: Read whole file: time dd if=FILE bs=FILESIZE of=/dev/null
T3,T4: Read last byte: time dd if=FILE bs=1 skip=FILESIZE-1 of=/dev/null
T1,T3: SATA SSD, raw read speed with dd bs=1M: ~520MB/s
T2,T4: USB3 flash drive via USB2, raw read speed: ~27MB/s
As expected, compression helps to improve 'virtual' read speed on slow
storage. Otherwise, it depends on storage speed, CPU speed, system load, ...
As unexpected (for me), even LZX seems to be suitable for random reads
which are done when EXE files are preloaded or paged-in.
If the files were already cached, all read times were similar: ~0.135s
for the whole file.
For more flexibility, I will provide a new version of the patch with
'--compact-os ALGORITHM' option.
Thanks,
Christian
