On 02/01/2022 16:33, Jorge P. de Morais Neto wrote:
Hi Piotr! Happy 2022!
Hi Jorge!
Likewise!
Have you revised after kernel Linux 4.15? The btrfs(5) manpage says:
Since kernel 4.15, a set of heuristic algorithms have been
improved by using frequency sampling, repeated pattern
detection and Shannon entropy calculation to avoid that.
Therefore, it looks like after kernel Linux 4.15 the compress option (or
compress=ALG:LEVEL), instead of compress-force, became more interesting.
Full paragraph says:
"Otherwise some simple heuristics are applied to detect an
incompressible file. If the first blocks written to a file are not
compressible, the whole file is permanently marked to skip compression.
As this is too simple, the compress-force is a workaround that will
compress most of the files at the cost of some wasted CPU cycles on
failed attempts. Since kernel 4.15, a set of heuristic algorithms have
been improved by using frequency sampling, repeated pattern detection
and Shannon entropy calculation to avoid that."
Meaning, compress has been improved to detect first block-uncompressible
rest-compressible files, and so on. But only improved.
Compress-force still remains 100% efficient in compressing everything,
for example 40GB VM disk file, where first 20GB is well used, but
remaining 20GB are empty with zeros, or zero-like patterns, which are
highly compressible. However "improved" compress is, I doubt it will
catch all these edge cases. I entrust compress-force to compress
everything for me without misses, at the cost of some CPU usage, most
likely much smaller CPU footprint than chosen compression level itself.
I am currently using compress-force=zstd:12 for the SSD and
compress=zstd:12 for both HDD (internal SATA and external USB3)¹.
Despite the strong compression level, performance is pretty good. Yet,
when the system settles, I intend to reduce compression level to 9 or 6
(as you earlier recommended). This should make performance even better,
while saving a lot of space because most data was compressed at
level 12.
I run compress-force=zstd:6 on my fast PC, compress-force=zstd:3 on my
server (to give it a bit more breathing space), and also same level 3 on
my laptop. 12 is quite high for SSD, are you sure you not slowing down
peak performance of your SSD by intense CPU usage? Or is it by design,
to reduce number of writes to SSD?
And I may also change compress-force to compress, even for the SSD,
because I run kernel 5.10 which is later than 4.15. I may ask the
linux-btrfs mailing list first. I have subscribed to it. Are you there
too?
No, I am not on this list, where is it? Please send a link!
Regards
* Footnotes
¹ Both HDD have compress (rather than compress-force) because most of
their files are already compressed---pictures, videos, music, compressed
archives etc.
Makes sense!
--
With kindest regards, Piotr.
⢀⣴⠾⠻⢶⣦⠀
⣾⠁⢠⠒⠀⣿⡁ Debian - The universal operating system
⢿⡄⠘⠷⠚⠋⠀ https://www.debian.org/
⠈⠳⣄⠀⠀⠀⠀
