Hi, Antonio Diaz Diaz wrote: > [email protected] wrote: >> lzip's initial memory check seems to be too conservative. Or does lzip >> uses more memory than xz and lzma_alone with the same dictionary size? > > IIRC lzip 1.11 does use a few MB more than lzma-utils for the same > dictionary size. I don't know the amount used by other lzma_alone > implementations. > > Lzip 1.11 does not perform any memory check. It tries to assign the > needed memory, and if it can't, it prints the "Not enough memory. Try a > smaller dictionary size." message. I think xz does perform a memory > check and adjusts the dictionary size accordingly.
xz doesn't perform a check by default. (You can specify --memlimit-compress= to tell it to do that.) > From http://www.nongnu.org/lzip/manual/lzip_manual.html#Introduction > "The amount of memory required for compression is about 5 MiB plus 1 or > 2 times the dictionary size limit (1 if input file size is less than > dictionary size limit, else 2) plus 8 times the dictionary size really > used. The option `-0' is special and only requires about 1.5 MiB at most." For a 256MiB dictionary then, lzip should have used about 2565MiB (compared to 2405MiB for a 240MiB dictionary). However, the lzip manual says: "Note that dictionary sizes are quantized. If the specified size does not match one of the valid sizes, it will be rounded upwards." What are the relevant sizes that the dictionary is quantized to? Could lzip have been trying to use a much larger dictionary size when I specified 256MiB? >> So for that specific file, lzip took about 2.5 times as long as xz. > > I guess your system is swapping slightly because of the large dictionary > size. As you can see, for example here[1], lzip and xz are comparable > both in compression time and size, with "lzip -9" tending to be slower > than "xz -9" but compressing a little more. > [1] http://martin-steigerwald.de/computer/programme/packbench/index.html > > >> Any idea what causes the time difference? Could >> different compiler options have that large an effect? I think I compiled >> xz with -O3 but lzip with -O2. > > I have only noticed small speed changes in lzip between -O2 and -O3. I > don't know the effects on xz, but I don't think they can be so large. The difference wasn't due to swapping. Can anyone else reproduce the difference I'm seeing? I tested lzip again, this time with a smaller, more compressible file (a tar archive, 633169920 bytes long). I used a 128MiB dictionary size to rule out the possibility of swapping being the cause of lzip being slower than xz. Again, there was a significant difference between the compression speed of lzip and xz, with lzip taking about 2.5 times as long as xz. The xz-compressed file was slightly smaller than the lzip one. $ time lzip --verbose --keep -9 --dictionary-size=128MiB file.tar file.tar: 1.891:1, 4.230 bits/byte, 47.12% saved, 633169920 in, 334811557 out. real 19m8.546s user 19m4.836s sys 0m3.296s $ time xz -v --compress --keep -9e --lzma2=dict=128MiB file.tar file.tar (1/1) 100 % 318.5 MiB / 603.8 MiB = 0.527 1.3 MiB/s 7:47 real 7m47.825s user 7m44.165s sys 0m2.760s $ du --bytes file.tar.xz 333996208 file.tar.xz _______________________________________________ Lzip-bug mailing list [email protected] http://lists.nongnu.org/mailman/listinfo/lzip-bug
