Hi Jim, thanks for the patch.
There are few problems with it though, that I've fixed.
More details below.
On Friday 09 December 2011, Jim Meyering wrote:
> Today I noticed that automake-generated Makefile's dist-xz rule
> used a hard-coded xz -9. That is a problem because on this front, xz
> differs from gzip and bzip2. While the latter two incur no run-time
> *decompression* penalty for using a higher compression level, with xz
> if you specify -9, that imposes a potentially fatal virtual-memory
> requirement on any client that wants to decompress your tar.xz file.
> People have complained that a tarball compressed with -9 cannot
> be uncompressed in a low-memory environment (wrt-based embedded).
>
> Hence, instead of defaulting to -9, which is useful only
> for very large tarballs, it defaults to -e (equivalent to -6e).
> This limits the default memory requirements imposed on decompressors,
> yet still gives very good compression ratios.
>
Thanks for the detailed explanation. Unfortunately, you have forgotten
to put it into the ChangeLog entry and git commit message, where IMO it
belongs; I've taken the liberty of doing so on your behalf.
> From 786d9b3e4ff4831dfea00b1010d24d23b399316e Mon Sep 17 00:00:00 2001
> From: Jim Meyering <meyer...@redhat.com>
> Date: Sat, 2 Oct 2010 22:30:02 +0200
> Subject: [PATCH] dist-xz, dist-bzip2: don't hard-code -9: honor envvar
> settings
>
> This is a cherry-pick of commit 6da46f31, with additional changes to
> reflect that the xz compression level should default to -e, not -9.
>
> * lib/am/distdir.am (dist-xz): Do not hard-code xz's -9: that made
> it impossible to override. Actually don't default to -9, either,
> since that induced inordinately large virtual memory usage when
> merely decompressing. Instead, use its XZ_OPT envvar, defaulting
> to -e if not defined. Suggested by Lasse Collin.
> (dist-bzip2): Similarly, do not hard-code -9, but do continue to
> use -9 by default. Honor the BZIP2 envvar.
> * NEWS (Miscellaneous changes): Mention it.
> * doc/automake.texi (The Types of Distributions): Describe the
> newly enabled environment variables.
> ---
> ChangeLog | 13 +++++++++++++
> NEWS | 5 +++++
> doc/automake.texi | 9 +++++++++
> lib/am/distdir.am | 4 ++--
> 4 files changed, 29 insertions(+), 2 deletions(-)
>
> diff --git a/ChangeLog b/ChangeLog
> index 1aba97a..41fa3bb 100644
> --- a/ChangeLog
> +++ b/ChangeLog
> @@ -1,3 +1,16 @@
> +2010-10-05 Jim Meyering <meyer...@redhat.com>
> +
Oops, wrong date. Fixed.
Also, I've added myself as secondary author, since I've contributed few
non-trivial amendings (see below). I hope you're ok with that.
> + dist-xz, dist-bzip2: don't hard-code -9: honor envvar settings
> + * lib/am/distdir.am (dist-xz): Do not hard-code xz's -9: that
> + made it impossible to override. Instead, use its XZ_OPT envvar,
> + defaulting to -9 if not defined. Thus no change in behavior
> + when XZ_OPT is not set, and now, this rule honors the setting
> + of that envvar when it is set. Suggested by Lasse Collin.
> + (dist-bzip2): Likewise for it's corresponding envvar: BZIP2.
> + * NEWS (Miscellaneous changes): Mention it.
> + * doc/automake.texi (The Types of Distributions): Describe the
> + newly enabled environment variables.
> +
>
Oops, you haven't update the ChangeLog entry, and it is not out-of-sync
with the code changes and the (correct) git commit message. I've fixed
this.
> 2011-12-07 Stefano Lattarini <stefano.lattar...@gmail.com>
>
> maint: sync auxiliary files from upstream
> diff --git a/NEWS b/NEWS
> index 6c85e1e..b2b5fb7 100644
> --- a/NEWS
> +++ b/NEWS
> @@ -31,6 +31,11 @@ New in 1.11.1a:
> - The `lzma' compression scheme and associated automake option `dist-lzma'
> is obsoleted by `xz' and `dist-xz' due to upstream changes.
>
> + - You may adjust the compression options used in dist-xz and dist-bzip2.
> + The default is now merely -e for xz, but still -9 for bzip; you may
> + specify a different level via the XZ_OPT and BZIP2 envvars respectively.
> + E.g., "make dist-xz XZ_OPT=-7" or "make dist-xz BZIP2=-5"
> +
> - The `compile' script now converts some options for MSVC for a better
> user experience. Similarly, the new `ar-lib' script wraps Microsoft lib.
>
> diff --git a/doc/automake.texi b/doc/automake.texi
> index 783463c..9797d92 100644
> --- a/doc/automake.texi
> +++ b/doc/automake.texi
> @@ -8674,9 +8674,13 @@ The Types of Distributions
> distributions in various formats. Their targets are:
>
> @table @asis
> +@vindex BZIP2
> @item @code{dist-bzip2}
> Generate a bzip2 tar archive of the distribution. bzip2 archives are
> frequently smaller than gzipped archives.
> +By default, this rule makes @samp{bzip2} use a compression option of
> @option{-9}.
> +To make it use a different one, set the @env{BZIP2} environment variable.
> +For example, @samp{make dist-bzip2 BZIP2=-7}.
> @trindex dist-bzip2
>
> @item @code{dist-gzip}
> @@ -8694,10 +8698,15 @@ The Types of Distributions
> Generate a shar archive of the distribution.
> @trindex dist-shar
>
> +@vindex XZ_OPT
> @item @code{dist-xz}
> Generate an @samp{xz} tar archive of the distribution. @command{xz}
> archives are frequently smaller than @command{bzip2}-compressed archives.
> The @samp{xz} format displaces the obsolete @samp{lzma} format.
> +By default, this rule makes @samp{xz} use a compression option of
> @option{-9}.
>
Nope, it uses a compression option of `-e', for the reasons you've so
clearly defended ;-) Fixed.
> +To make it use a different one, set the @env{XZ_OPT} environment variable.
> +For example, run this command to use the default compression ratio, but
> +with a progress indicator: @samp{make dist-xz XZ_OPT=-7e}.
> @trindex dist-xz
>
>
> diff --git a/lib/am/distdir.am b/lib/am/distdir.am
> index 41ff14a..65cb668 100644
> --- a/lib/am/distdir.am
> +++ b/lib/am/distdir.am
> @@ -341,7 +341,7 @@ dist-gzip: distdir
> ?BZIP2?DIST_ARCHIVES += $(distdir).tar.bz2
> .PHONY: dist-bzip2
> dist-bzip2: distdir
> - tardir=$(distdir) && $(am__tar) | bzip2 -9 -c >$(distdir).tar.bz2
> + tardir=$(distdir) && $(am__tar) | BZIP2=$${BZIP2--9} bzip2 -c
> >$(distdir).tar.bz2
> $(am__remove_distdir)
>
> ?LZMA?DIST_ARCHIVES += $(distdir).tar.lzma
> @@ -353,7 +353,7 @@ dist-lzma: distdir
> ?XZ?DIST_ARCHIVES += $(distdir).tar.xz
> .PHONY: dist-xz
> dist-xz: distdir
> - tardir=$(distdir) && $(am__tar) | xz -9 -c >$(distdir).tar.xz
> + tardir=$(distdir) && $(am__tar) | XZ_OPT=$${XZ_OPT--e} xz -c
> >$(distdir).tar.xz
> $(am__remove_distdir)
>
> ?COMPRESS?DIST_ARCHIVES += $(distdir).tar.Z
>
You have forgotten to update the recipes for `dist' and `dist-all':
$ grep -e -9 lib/am/distdir.am
tardir=$(distdir) && $(am__tar) | BZIP2=$${BZIP2--9} bzip2 -c
>$(distdir).tar.bz2
tardir=$(distdir) && $(am__tar) | lzma -9 -c >$(distdir).tar.lzma
?BZIP2? tardir=$(distdir) && $(am__tar) | bzip2 -9 -c >$(distdir).tar.bz2
?LZMA? tardir=$(distdir) && $(am__tar) | lzma -9 -c >$(distdir).tar.lzma
?XZ? tardir=$(distdir) && $(am__tar) | xz -9 -c >$(distdir).tar.xz
I've fixed this "the dumb way", i.e., by copying & pasting. Extra
points will go to anyone which comes up with a refactoring patch that
removes this annoying code duplication.
And a final nit: you have forgotten to run ./boostrap to regenerate
Automake's own Makefile.in (sorry, theis is still version-controlled
for now!). I've fixed that as well.
Attached is what I've pushed to maint (and merged into branch-1.11).
Thanks,
Stefano
From c8e01d581a7e7c2445a139def46939a547951746 Mon Sep 17 00:00:00 2001
Message-Id: <c8e01d581a7e7c2445a139def46939a547951746.1323473268.git.stefano.lattar...@gmail.com>
From: Jim Meyering <meyer...@redhat.com>
Date: Fri, 9 Dec 2011 23:17:18 +0100
Subject: [PATCH] dist-xz, dist-bzip2: don't hard-code -9, honor envvar settings
Before the present change, automake-generated `dist-xz' rule used
a hard-coded `xz -9'. That was a problem because on this front,
xz differs from gzip and bzip2. While the latter two don't incur
any run-time decompression penalty for using a higher compression
level, specifying -9 with xz imposes a potentially fatal virtual
memory requirement on any client that wants to decompress your
tar.xz file.
People have complained that a tarball compressed with -9 cannot
be uncompressed in a low-memory environment (wrt-based embedded).
Hence, instead of defaulting to -9, which is useful only for very
large tarballs, it defaults to -e (equivalent to -6e). This
limits the default memory requirements imposed on decompressors,
yet still gives very good compression ratios.
* lib/am/distdir.am (dist-xz): Do not hard-code xz's -9: that made
it impossible to override. Actually don't default to -9, either,
since that induced inordinately large virtual memory usage when
merely decompressing. Instead, use its XZ_OPT envvar, defaulting
to -e if not defined. Suggested by Lasse Collin.
(dist, dist-all) [?XZ?]: Likewise
(dist-bzip2): Similarly, do not hard-code -9, but do continue to
use -9 by default. Honor the BZIP2 envvar.
(dist, dist-all) [?BZIP2?]: Likewise
* NEWS: Update.
* doc/automake.texi (The Types of Distributions): Describe the
newly enabled environment variables.
This is inspired to commit v1.11-389-g6da46f3, with additional
changes to reflect that the xz compression level should default
to -e, not -9.
---
ChangeLog | 30 ++++++++++++++++++++++++++++++
Makefile.in | 6 +++---
NEWS | 7 +++++--
doc/automake.texi | 10 ++++++++++
lib/am/distdir.am | 8 ++++----
5 files changed, 52 insertions(+), 9 deletions(-)
diff --git a/ChangeLog b/ChangeLog
index e13e833..618ab02 100644
--- a/ChangeLog
+++ b/ChangeLog
@@ -1,3 +1,33 @@
+2011-12-09 Jim Meyering <meyer...@redhat.com>
+ Stefano Lattarini <stefano.lattar...@gmail.com>
+
+ dist-xz, dist-bzip2: don't hard-code -9, honor envvar settings
+ Before the present change, automake-generated `dist-xz' rule used
+ a hard-coded `xz -9'. That was a problem because on this front,
+ xz differs from gzip and bzip2. While the latter two don't incur
+ any run-time decompression penalty for using a higher compression
+ level, specifying -9 with xz imposes a potentially fatal virtual
+ memory requirement on any client that wants to decompress your
+ tar.xz file.
+ People have complained that a tarball compressed with -9 cannot
+ be uncompressed in a low-memory environment (wrt-based embedded).
+ Hence, instead of defaulting to -9, which is useful only for very
+ large tarballs, it defaults to -e (equivalent to -6e). This
+ limits the default memory requirements imposed on decompressors,
+ yet still gives very good compression ratios.
+ * lib/am/distdir.am (dist-xz): Do not hard-code xz's -9: that made
+ it impossible to override. Actually don't default to -9, either,
+ since that induced inordinately large virtual memory usage when
+ merely decompressing. Instead, use its XZ_OPT envvar, defaulting
+ to -e if not defined. Suggested by Lasse Collin.
+ (dist, dist-all) [?XZ?]: Likewise
+ (dist-bzip2): Similarly, do not hard-code -9, but do continue to
+ use -9 by default. Honor the BZIP2 envvar.
+ (dist, dist-all) [?BZIP2?]: Likewise
+ * NEWS: Update.
+ * doc/automake.texi (The Types of Distributions): Describe the
+ newly enabled environment variables.
+
2011-12-09 Stefano Lattarini <stefano.lattar...@gmail.com>
* NEWS: Fix typos, grammaros and suboptimal wording.
diff --git a/Makefile.in b/Makefile.in
index 9a03035..e475693 100644
--- a/Makefile.in
+++ b/Makefile.in
@@ -671,7 +671,7 @@ dist-gzip: distdir
tardir=$(distdir) && $(am__tar) | GZIP=$(GZIP_ENV) gzip -c >$(distdir).tar.gz
$(am__remove_distdir)
dist-bzip2: distdir
- tardir=$(distdir) && $(am__tar) | bzip2 -9 -c >$(distdir).tar.bz2
+ tardir=$(distdir) && $(am__tar) | BZIP2=$${BZIP2--9} bzip2 -c >$(distdir).tar.bz2
$(am__remove_distdir)
dist-lzma: distdir
@@ -679,7 +679,7 @@ dist-lzma: distdir
$(am__remove_distdir)
dist-xz: distdir
- tardir=$(distdir) && $(am__tar) | xz -9 -c >$(distdir).tar.xz
+ tardir=$(distdir) && $(am__tar) | XZ_OPT=$${XZ_OPT--e} xz -c >$(distdir).tar.xz
$(am__remove_distdir)
dist-tarZ: distdir
@@ -697,7 +697,7 @@ dist-zip: distdir
dist dist-all: distdir
tardir=$(distdir) && $(am__tar) | GZIP=$(GZIP_ENV) gzip -c >$(distdir).tar.gz
- tardir=$(distdir) && $(am__tar) | bzip2 -9 -c >$(distdir).tar.bz2
+ tardir=$(distdir) && $(am__tar) | BZIP2=$${BZIP2--9} bzip2 -c >$(distdir).tar.bz2
$(am__remove_distdir)
# This target untars the dist file and tries a VPATH configuration. Then
diff --git a/NEWS b/NEWS
index d40dc26..790fabf 100644
--- a/NEWS
+++ b/NEWS
@@ -27,6 +27,11 @@ New in 1.11.0a:
- The `lzma' compression scheme and associated automake option `dist-lzma'
is obsoleted by `xz' and `dist-xz' due to upstream changes.
+ - You may adjust the compression options used in dist-xz and dist-bzip2.
+ The default is now merely -e for xz, but still -9 for bzip; you may
+ specify a different level via the XZ_OPT and BZIP2 envvars respectively.
+ E.g., "make dist-xz XZ_OPT=-7" or "make dist-xz BZIP2=-5"
+
- The py-compile script now accepts empty arguments passed to the options
`--destdir' and `--basedir', and complains about unrecognized options.
Moreover, a non-option argument or a special `--' argument terminates
@@ -69,8 +74,6 @@ Bugs fixed in 1.11.0a:
not used, `make' output no longer contains spurious backslash-only
lines, thus once again matching what Automake did before 1.11.
- - The `dist-xz' option now uses `xz -9' for maximum compression.
-
- The AM_COND_IF macro also works if the shell expression for the conditional
is no longer valid for the condition.
diff --git a/doc/automake.texi b/doc/automake.texi
index f6039bb..a5f857d 100644
--- a/doc/automake.texi
+++ b/doc/automake.texi
@@ -8645,9 +8645,13 @@ Automake generates rules to provide archives of the project for
distributions in various formats. Their targets are:
@table @asis
+@vindex BZIP2
@item @code{dist-bzip2}
Generate a bzip2 tar archive of the distribution. bzip2 archives are
frequently smaller than gzipped archives.
+By default, this rule makes @samp{bzip2} use a compression option of
+@option{-9}. To make it use a different one, set the @env{BZIP2}
+environment variable. For example, @samp{make dist-bzip2 BZIP2=-7}.
@trindex dist-bzip2
@item @code{dist-gzip}
@@ -8665,10 +8669,16 @@ instead.
Generate a shar archive of the distribution.
@trindex dist-shar
+@vindex XZ_OPT
@item @code{dist-xz}
Generate an @samp{xz} tar archive of the distribution. @command{xz}
archives are frequently smaller than @command{bzip2}-compressed archives.
The @samp{xz} format displaces the obsolete @samp{lzma} format.
+By default, this rule makes @samp{xz} use a compression option of
+@option{-e}. To make it use a different one, set the @env{XZ_OPT}
+environment variable. For example, run this command to use the
+default compression ratio, but with a progress indicator:
+@samp{make dist-xz XZ_OPT=-7e}.
@trindex dist-xz
@item @code{dist-zip}
diff --git a/lib/am/distdir.am b/lib/am/distdir.am
index 41ff14a..f685ec1 100644
--- a/lib/am/distdir.am
+++ b/lib/am/distdir.am
@@ -341,7 +341,7 @@ dist-gzip: distdir
?BZIP2?DIST_ARCHIVES += $(distdir).tar.bz2
.PHONY: dist-bzip2
dist-bzip2: distdir
- tardir=$(distdir) && $(am__tar) | bzip2 -9 -c >$(distdir).tar.bz2
+ tardir=$(distdir) && $(am__tar) | BZIP2=$${BZIP2--9} bzip2 -c >$(distdir).tar.bz2
$(am__remove_distdir)
?LZMA?DIST_ARCHIVES += $(distdir).tar.lzma
@@ -353,7 +353,7 @@ dist-lzma: distdir
?XZ?DIST_ARCHIVES += $(distdir).tar.xz
.PHONY: dist-xz
dist-xz: distdir
- tardir=$(distdir) && $(am__tar) | xz -9 -c >$(distdir).tar.xz
+ tardir=$(distdir) && $(am__tar) | XZ_OPT=$${XZ_OPT--e} xz -c >$(distdir).tar.xz
$(am__remove_distdir)
?COMPRESS?DIST_ARCHIVES += $(distdir).tar.Z
@@ -395,9 +395,9 @@ endif %?SUBDIRS%
dist dist-all: distdir
?GZIP? tardir=$(distdir) && $(am__tar) | GZIP=$(GZIP_ENV) gzip -c >$(distdir).tar.gz
-?BZIP2? tardir=$(distdir) && $(am__tar) | bzip2 -9 -c >$(distdir).tar.bz2
+?BZIP2? tardir=$(distdir) && $(am__tar) | BZIP2=$${BZIP2--9} bzip2 -c >$(distdir).tar.bz2
?LZMA? tardir=$(distdir) && $(am__tar) | lzma -9 -c >$(distdir).tar.lzma
-?XZ? tardir=$(distdir) && $(am__tar) | xz -9 -c >$(distdir).tar.xz
+?XZ? tardir=$(distdir) && $(am__tar) | XZ_OPT=$${XZ_OPT--e} xz -c >$(distdir).tar.xz
?COMPRESS? tardir=$(distdir) && $(am__tar) | compress -c >$(distdir).tar.Z
?SHAR? shar $(distdir) | GZIP=$(GZIP_ENV) gzip -c >$(distdir).shar.gz
?ZIP? -rm -f $(distdir).zip
--
1.7.2.3
