Bastian Blank: > On Mon, Jan 05, 2015 at 06:56:10PM +0100, Jérémy Bobbio wrote: > > > > > * linux-source: mtimes of many files differ. Would it be ok to just > > > > > create the tarball with a single timestamp (`tar --mtime=`)? > > I've used find+touch instead. See attached patch. > > Why?
Good question. It's the way we are doing it in dh_builddeb. But for
kernel-source, using `--mtime` is more straight forward. This is done in
the attached patch.
> > I also stumbled on a variation of the Installed-Size field in one of the
> > .deb. But this is a different topic.
>
> Is this value computed in a stable way?
The value can differ greatly depending on the underlying filesystem.
This is to be addressed at dpkg's level. josch is working on it.
> > +@@ -301,7 +301,9 @@ if [ ! -z ${output_file} ]; then
> > + if [ -z ${cpio_file} ]; then
> > + timestamp=
> > + if test -n "$KBUILD_BUILD_TIMESTAMP"; then
> > +- timestamp="$(date -d"$KBUILD_BUILD_TIMESTAMP" +%s || :)"
> > ++ source_date=$(echo "$KBUILD_BUILD_TIMESTAMP" |
> > ++ sed -e 's/.*(\([0-9-]\+\)).*/\1/')
> > ++ timestamp="$(date -d"$source_date" +%s || :)"
>
> This solution may not work. The patched source can be built with a
> normal timestamp override, via linux-source.
The above construction will work given a standard date in
KBUILD_BUILD_TIMESTAMP. The sed expression only match parenthesises.
I am open to other suggestions.
> > @@ -87,6 +87,8 @@
> > rm -rf '$@' '$(DIR)'
> > $(call copy_source,$(DIR))
> > chmod -R u+rw,go=rX '$(DIR)'
> > + find '$(DIR)' -depth -newermt '$(SOURCE_DATE)' -print0 | \
> > + xargs -0r touch --no-dereference --date='$(SOURCE_DATE)'
>
> -newermt? This does not look really stable.
I don't understand what you mean by that. It's been proven working for
a while in our experiments.
With the attached patch, linux can be built reproducibly with the
current experimental framework. The following changes have been
introduced since the previous patch:
* Files in the source tarball are added in a stable order.
* kernel-doc is made deterministic when parsing `@foo()` in
descriptions.
* The last set of manpages when generating manpages from files
referenced in multiple sections of the documentation is
deterministic.
Some `@foo()` constructs in the kernel should probably be fixed to be
just `foo()`. It's unclear what should be used for functions passed as
parameters.
--
Lunar .''`.
lu...@debian.org : :Ⓐ : # apt-get install anarchism
`. `'`
`-
diff -Nru linux-3.16.7-ckt2/debian/changelog linux-3.16.7-ckt2/debian/changelog
--- linux-3.16.7-ckt2/debian/changelog 2014-12-08 21:03:19.000000000 +0100
+++ linux-3.16.7-ckt2/debian/changelog 2015-01-07 16:50:40.000000000 +0100
@@ -1,3 +1,16 @@
+linux (3.16.7-ckt2-1.0~reproducible1) UNRELEASED; urgency=low
+
+ * Make build reproducible:
+ - Add a patch to make gen_initramfs.sh parse Debian's KBUILD_BUILD_TIMESTAMP.
+ - Add a patch to make kernel-doc parse highlights determistically.
+ - Add a patch to force the order when generating manpages from files
+ referenced in multiple sections of the documentation.
+ - Use debian/changelog date as mtimes in the generated source tarball.
+ - Write files in the source tarball in a stable order.
+ - Stop recording current time in gzip headers when compressing documentation.
+
+ -- Jérémy Bobbio <lu...@debian.org> Wed, 07 Jan 2015 16:50:39 +0100
+
linux (3.16.7-ckt2-1) unstable; urgency=high
* New upstream stable update:
diff -Nru linux-3.16.7-ckt2/debian/patches/debian/generate-manpage-in-deterministic-order.patch linux-3.16.7-ckt2/debian/patches/debian/generate-manpage-in-deterministic-order.patch
--- linux-3.16.7-ckt2/debian/patches/debian/generate-manpage-in-deterministic-order.patch 1970-01-01 01:00:00.000000000 +0100
+++ linux-3.16.7-ckt2/debian/patches/debian/generate-manpage-in-deterministic-order.patch 2015-01-07 16:34:00.000000000 +0100
@@ -0,0 +1,55 @@
+Description: generate manpages in deterministic order
+ When header files are referenced multiple times in different
+ documentation parts, manpages for the various functions will also
+ be generated multiple times.
+ .
+ In order to support reproducible builds when multiple jobs are
+ enabled, we need to make the order deterministic. To do so, we
+ scan the documentation, identify sections having similar directives,
+ and generate dependencies making the first section in alphabetical
+ order build last.
+ .
+ With the current kernel, the following dependencies are generated:
+ .
+ Documentation/DocBook/alsa-driver-api.tmpl: Documentation/DocBook/device-drivers.tmpl
+ Documentation/DocBook/gadget.tmpl: Documentation/DocBook/usb.tmpl
+ Documentation/DocBook/genericirq.tmpl: Documentation/DocBook/kernel-api.tmpl
+Author: Jérémy Bobbio <lu...@debian.org>
+
+--- linux-3.16.7-ckt2.orig/Documentation/DocBook/Makefile
++++ linux-3.16.7-ckt2/Documentation/DocBook/Makefile
+@@ -92,6 +92,24 @@ endef
+ %.xml: %.tmpl $(KERNELDOC) $(DOCPROC) FORCE
+ $(call if_changed_rule,docproc)
+
++# Ensure a deterministic build order when the same file are sourced in
++# different locations
++TEMPLATES = $(patsubst %.xml,%.tmpl,$(BOOKS))
++$(obj)/.docbook.d: $(TEMPLATES)
++ @set -e; \
++ cd $(srctree); \
++ grep '^!' $(TEMPLATES) | \
++ tr ':' '\t' | LC_ALL=C sort -k2 | \
++ awk ' { if (previous_include == $$2 && previous_source != $$1) { \
++ print previous_source ": " $$1; \
++ } else { \
++ previous_source = $$1; \
++ previous_include = $$2; \
++ } }' | sort -u | \
++ sed -e 's/\.tmpl/\.9/g' > $@
++
++-include $(obj)/.docbook.d
++
+ # Tell kbuild to always build the programs
+ always := $(hostprogs-y)
+
+@@ -218,7 +236,8 @@ clean-files := $(DOCBOOKS) \
+ $(patsubst %.xml, %.pdf, $(DOCBOOKS)) \
+ $(patsubst %.xml, %.html, $(DOCBOOKS)) \
+ $(patsubst %.xml, %.9, $(DOCBOOKS)) \
+- $(index)
++ $(index) \
++ $(obj)/.docbook.d
+
+ clean-dirs := $(patsubst %.xml,%,$(DOCBOOKS)) man
+
diff -Nru linux-3.16.7-ckt2/debian/patches/debian/make-highlights-deterministic-in-kernel-doc.patch linux-3.16.7-ckt2/debian/patches/debian/make-highlights-deterministic-in-kernel-doc.patch
--- linux-3.16.7-ckt2/debian/patches/debian/make-highlights-deterministic-in-kernel-doc.patch 1970-01-01 01:00:00.000000000 +0100
+++ linux-3.16.7-ckt2/debian/patches/debian/make-highlights-deterministic-in-kernel-doc.patch 2015-01-07 16:43:31.000000000 +0100
@@ -0,0 +1,21 @@
+Description: parse kernel-doc deterministically
+ Regular expressions for highlights in kernel-doc are stored in a Perl
+ hash. These hashes are ordered differently for each Perl run. This will
+ prevent kernel-doc to behave deterministically when parsing
+ “@foo()” as in some runs it will be interpreted as a parameter and
+ in the others it will be interpreted as a function.
+ .
+ We now sort the %highlights hash to get the same behavior on every run.
+Author: Jérémy Bobbio <lu...@debian.org>
+
+--- linux-3.16.7-ckt2.orig/scripts/kernel-doc
++++ linux-3.16.7-ckt2/scripts/kernel-doc
+@@ -2586,7 +2586,7 @@ $kernelversion = get_kernel_version();
+
+ # generate a sequence of code that will splice in highlighting information
+ # using the s// operator.
+-foreach my $pattern (keys %highlights) {
++foreach my $pattern (sort keys %highlights) {
+ # print STDERR "scanning pattern:$pattern, highlight:($highlights{$pattern})\n";
+ $dohighlight .= "\$contents =~ s:$pattern:$highlights{$pattern}:gs;\n";
+ }
diff -Nru linux-3.16.7-ckt2/debian/patches/debian/parse-debian-build-timestamp.patch linux-3.16.7-ckt2/debian/patches/debian/parse-debian-build-timestamp.patch
--- linux-3.16.7-ckt2/debian/patches/debian/parse-debian-build-timestamp.patch 1970-01-01 01:00:00.000000000 +0100
+++ linux-3.16.7-ckt2/debian/patches/debian/parse-debian-build-timestamp.patch 2015-01-05 14:50:58.000000000 +0100
@@ -0,0 +1,26 @@
+Description: parse Debian KBUILD_BUILD_TIMESTAMP in gen_initramfs_list.sh
+ gen_initramfs_list.sh assumes that KBUILD_BUILD_TIMESTAMP only contains a
+ date. In Debian, it's a longer string looking like:
+ “Debian 3.16.7-2 (2014-11-06)”. Here we adapt the script to parse Debian
+ format.
+ .
+ This change is necessary to have reproducible builds as the date will
+ then be used as mtime for files in initramfs archive instead of the
+ current time.
+Author: Jérémy Bobbio <lu...@debian.org>
+Bug-Debian: https://bugs.debian.org/769844
+Forwarded: not-needed
+
+--- linux-3.16.7-ckt2.orig/scripts/gen_initramfs_list.sh
++++ linux-3.16.7-ckt2/scripts/gen_initramfs_list.sh
+@@ -301,7 +301,9 @@ if [ ! -z ${output_file} ]; then
+ if [ -z ${cpio_file} ]; then
+ timestamp=
+ if test -n "$KBUILD_BUILD_TIMESTAMP"; then
+- timestamp="$(date -d"$KBUILD_BUILD_TIMESTAMP" +%s || :)"
++ source_date=$(echo "$KBUILD_BUILD_TIMESTAMP" |
++ sed -e 's/.*(\([0-9-]\+\)).*/\1/')
++ timestamp="$(date -d"$source_date" +%s || :)"
+ if test -n "$timestamp"; then
+ timestamp="-t $timestamp"
+ fi
diff -Nru linux-3.16.7-ckt2/debian/patches/series linux-3.16.7-ckt2/debian/patches/series
--- linux-3.16.7-ckt2/debian/patches/series 2014-12-08 21:00:20.000000000 +0100
+++ linux-3.16.7-ckt2/debian/patches/series 2015-01-07 16:34:21.000000000 +0100
@@ -7,6 +7,9 @@
debian/arch-sh4-fix-uimage-build.patch
debian/powerpcspe-omit-uimage.patch
features/all/Kbuild-kconfig-Verbose-version-of-listnewconfig.patch
+debian/parse-debian-build-timestamp.patch
+debian/make-highlights-deterministic-in-kernel-doc.patch
+debian/generate-manpage-in-deterministic-order.patch
# Fixes/improvements to firmware loading
features/all/drivers-media-dvb-usb-af9005-request_firmware.patch
--- linux-3.16.7-ckt2/debian/rules.real 2014-11-04 05:41:34.000000000 +0100
+++ linux-3.16.7-ckt2/debian/rules.real 2015-01-07 14:37:52.000000000 +0100
@@ -87,7 +87,9 @@
rm -rf '$@' '$(DIR)'
$(call copy_source,$(DIR))
chmod -R u+rw,go=rX '$(DIR)'
- cd '$(BUILD_DIR)'; tar -caf 'linux-source-$(UPSTREAMVERSION).tar.xz' 'linux-source-$(UPSTREAMVERSION)'
+ cd '$(BUILD_DIR)'; find 'linux-source-$(UPSTREAMVERSION)' -type f -print0 | \
+ LC_ALL=C sort -z | tar --mtime='$(SOURCE_DATE)' --null -T - \
+ -caf 'linux-source-$(UPSTREAMVERSION).tar.xz'
rm -rf '$(DIR)'
$(BUILD_DIR)/linux-patch-$(UPSTREAMVERSION)-%.patch.xz: $(STAMPS_DIR)/source_none $(STAMPS_DIR)/source_%
@@ -202,7 +204,7 @@
cd $(DIR)/Documentation/DocBook; \
find * -name '*.html' -print | \
cpio -pd --preserve-modification-time '$(CURDIR)/$(OUT_DIR)/html'
- gzip -9qfr $(OUT_DIR)/Documentation
+ gzip -9nqfr $(OUT_DIR)/Documentation
+$(MAKE_SELF) install-base
install-manual: PACKAGE_NAME = linux-manual-$(VERSION)
signature.asc
Description: Digital signature
