>> If anyone is interested I can prepare an appropriate patch. Just let me know. > > Mikołaj, > > I am interested in a patch.
I am about to implement this solution in a newer version of lbzip2. I may be able to backport the patch to 0.23. > I looked over lbzip2 code enough to realize I couldn't think of a simple > patch for this issue. I was thinking it would be simple to limit the > number of memory buffers between worker and muxer. Then looking at how > blocks are split into input for the workers, started to worry that > this would get into trouble with deadlocks on blocks which are split > between two input blocks. So I filed a bug with no patch. ;) > > My guess of the easiest way to "fix" this is to change the behavior > of the splitter. Instead of sending random blocks of input to the > worker threads for them to split, the splitter actually reads and scans > for the block markers at the same time and passes along single blocks into > a queue of worker threads. Then a worker who is picking up a new block > to decompress can check if he is "too far ahead" of the muxer and then > block/wait for the muxer to catch up. I don't think this would scale well. The scanner implemented in lbzip2 0.23 is naive and slow. As Laszlo said, splitter would be the bottleneck. I am currently working on lbzip2 2.0, which is going to include some new features, among which there is a command-line option for limiting max memory usage during decompression. I hope lbzip2 2.0 can be released within a month. > On Thu, 20 Oct 2011 20:23:05 +0200 (CEST) "Ersek, Laszlo" > <la...@caesar.elte.hu> wrote: >> I'll tell you this: you might want to check out pbzip2; I believe >> pbzip2 should have a working memory throttle staring with v1.1.2. > > Laszlo, > > Yes, thanks for the suggestion. I do fallback on pbzip2 when lbzip2 fails. > > I used to use pbzip2, back then decompression required it to read the > input file twice, first pass was single threaded and scanned for block > markers, the second read was feed to worker threads to decompress... for > large input streams (anything larger than fit in ram) that implementation > sucked as it was input I/O bound for many minutes before generating > any output. This double read also prevented it from decompressing pipes. > > At that time, I think due to the double I/O, lbzip2 was wall clock faster > for any given input stream. I see now that the pbzip2 I/O layer has > changed and it appears to support pipes and scans input only once. > > I should re-compare them. The problem with pbzip2 splitter is that (AFAIK) it recognizes stream boundaries only (bz2 files consist of streams, streams consist of blocks). That's no problem if you decompress files created by lbzip2 <= 0.23 or pbzip2. Again, you should probably wait for lbzip2-2.0. Wall clock times are usually much better compared to 0.23 (or pbzip2), for example on my 4 core amd64 desktop: $ time lbzip2-2.0 -dkf linux-3.0.4.tar.bz2 real 9.03 user 29.78 sys 1.61 $ time lbzip2-0.23 -dkf linux-3.0.4.tar.bz2 real 16.29 user 53.90 sys 1.92 -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
