it would have been **much** faster if I had use the -m flag and set a higher memory ( in ram) amount.
On Mon, Jun 16, 2014 at 4:11 PM, andrew mcelroy <[email protected]> wrote: > > > > On Mon, Jun 16, 2014 at 4:06 PM, Jack Coats <[email protected]> wrote: > >> Sow how long did it take for how many fiels? >> > > $ tar -c XXXX | pbzip2 -vc > XXXX.tar.bz2 > Parallel BZIP2 v1.1.8 - by: Jeff Gilchrist [http://compression.ca] > [Jun. 10, 2012] (uses libbzip2 by Julian Seward) > Major contributions: Yavor Nikolov <[email protected]> > > # CPUs: 4 > BWT Block Size: 900 KB > File Block Size: 900 KB > Maximum Memory: 100 MB > ------------------------------------------- > File #: 1 of 1 > Input Name: <stdin> > Output Name: <stdout> > > Compressing data... > Output Size: 205714360 bytes > ------------------------------------------- > > Wall Clock: 5115.167256 seconds > > Don't let the files 1 of 1 fool you, that was because it was receiving > input from 1 descriptor (tar). > > This was on a 4 core i7 2.7 GHZ iMac. > > Respectfully, > Andrew McElroy > > > > > >> >> >> On Mon, Jun 16, 2014 at 2:27 PM, andrew mcelroy <[email protected]> >> wrote: >> >>> I found myself having to compress a million files less than 12k in size >>> each. >>> tar was likely to take the better part of a full working day (possibly >>> longer, I killed the process and didn't use pv (another great asleep at the >>> prompt topic). >>> >>> Enter: pbzip2 >>> http://compression.ca/pbzip2/ >>> >>> I'll let the website do it's own talking. It's a parallel bzip2 >>> implementation which can auto detect the number of cores it has to work >>> with. Fantastic stuff. >>> >>> Respectfully, >>> Andrew McElroy >>> >>> -- >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "NLUG" group. >>> To post to this group, send email to [email protected] >>> To unsubscribe from this group, send email to >>> [email protected] >>> For more options, visit this group at >>> http://groups.google.com/group/nlug-talk?hl=en >>> >>> --- >>> You received this message because you are subscribed to the Google >>> Groups "NLUG" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> >> >> -- >> ><> ... Jack >> >> "Whatever you do, work at it with all your heart"... Colossians 3:23 >> "If you are not part of the solution, you are part of the precipitate" - >> Henry J. Tillman >> "Anyone who has never made a mistake, has never tried anything new." - >> Albert Einstein >> "You don't manage people; you manage things. You lead people." - Admiral >> Grace Hopper, USN >> "a nanosecond is the time it takes electrons to propigate 11.8 inches" - >> " - http://youtu.be/JEpsKnWZrJ8 >> "Life is complex: it has a real part and an imaginary part." - Martin >> Terma >> >> -- >> -- >> You received this message because you are subscribed to the Google Groups >> "NLUG" group. >> To post to this group, send email to [email protected] >> To unsubscribe from this group, send email to >> [email protected] >> For more options, visit this group at >> http://groups.google.com/group/nlug-talk?hl=en >> >> --- >> You received this message because you are subscribed to the Google Groups >> "NLUG" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> For more options, visit https://groups.google.com/d/optout. >> > > -- -- You received this message because you are subscribed to the Google Groups "NLUG" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/nlug-talk?hl=en --- You received this message because you are subscribed to the Google Groups "NLUG" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
