On Mon, Jun 16, 2014 at 4:06 PM, Jack Coats <[email protected]> wrote: > Sow how long did it take for how many fiels? >
$ tar -c XXXX | pbzip2 -vc > XXXX.tar.bz2 Parallel BZIP2 v1.1.8 - by: Jeff Gilchrist [http://compression.ca] [Jun. 10, 2012] (uses libbzip2 by Julian Seward) Major contributions: Yavor Nikolov <[email protected]> # CPUs: 4 BWT Block Size: 900 KB File Block Size: 900 KB Maximum Memory: 100 MB ------------------------------------------- File #: 1 of 1 Input Name: <stdin> Output Name: <stdout> Compressing data... Output Size: 205714360 bytes ------------------------------------------- Wall Clock: 5115.167256 seconds Don't let the files 1 of 1 fool you, that was because it was receiving input from 1 descriptor (tar). This was on a 4 core i7 2.7 GHZ iMac. Respectfully, Andrew McElroy > > > On Mon, Jun 16, 2014 at 2:27 PM, andrew mcelroy <[email protected]> > wrote: > >> I found myself having to compress a million files less than 12k in size >> each. >> tar was likely to take the better part of a full working day (possibly >> longer, I killed the process and didn't use pv (another great asleep at the >> prompt topic). >> >> Enter: pbzip2 >> http://compression.ca/pbzip2/ >> >> I'll let the website do it's own talking. It's a parallel bzip2 >> implementation which can auto detect the number of cores it has to work >> with. Fantastic stuff. >> >> Respectfully, >> Andrew McElroy >> >> -- >> -- >> You received this message because you are subscribed to the Google Groups >> "NLUG" group. >> To post to this group, send email to [email protected] >> To unsubscribe from this group, send email to >> [email protected] >> For more options, visit this group at >> http://groups.google.com/group/nlug-talk?hl=en >> >> --- >> You received this message because you are subscribed to the Google Groups >> "NLUG" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> For more options, visit https://groups.google.com/d/optout. >> > > > > -- > ><> ... Jack > > "Whatever you do, work at it with all your heart"... Colossians 3:23 > "If you are not part of the solution, you are part of the precipitate" - > Henry J. Tillman > "Anyone who has never made a mistake, has never tried anything new." - > Albert Einstein > "You don't manage people; you manage things. You lead people." - Admiral > Grace Hopper, USN > "a nanosecond is the time it takes electrons to propigate 11.8 inches" - " > - http://youtu.be/JEpsKnWZrJ8 > "Life is complex: it has a real part and an imaginary part." - Martin Terma > > -- > -- > You received this message because you are subscribed to the Google Groups > "NLUG" group. > To post to this group, send email to [email protected] > To unsubscribe from this group, send email to > [email protected] > For more options, visit this group at > http://groups.google.com/group/nlug-talk?hl=en > > --- > You received this message because you are subscribed to the Google Groups > "NLUG" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > For more options, visit https://groups.google.com/d/optout. > -- -- You received this message because you are subscribed to the Google Groups "NLUG" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/nlug-talk?hl=en --- You received this message because you are subscribed to the Google Groups "NLUG" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
