Re: Repeatable builds

[Christopher Schultz]

Mark,

On 3/22/22 15:30, Mark Thomas wrote:

On 22/03/2022 19:06, Christopher Schultz wrote:

On 3/22/22 13:12, Mark Thomas wrote:




The JSign Ant task that adds the detached signature doesn't close the 
signed file. This causes problems for Ant. I've opened a JSign issue 
[2] for this. I have a locally build version with a hack that fixes 
the issue so I can continue testing.


Do you mean it doesn't call OutputStream.close()? Shouldn't that be 
cleaned up when the process exits?


Effectively, yes. The problem is that the Ant task runs in process so 
the file is still open when the next Ant target tries to work with the 
file. You won't notice the issue running JSign from the command line.


Can we run the jsign task with fork="true" or is there too much 
information in the ant process's memory that would need to be 
insecurely-sent to the jsign process to be prudent?


If it were just the full docs package, I don't think I'd worry too 
much about the Javadoc issue but the Windows installer is more of a 
problem. Therefore, I plan to work on a custom Ant task that will fix 
these zip files by setting the timestamps on the compressed files to 
be the same as the timestamp used by the rest of the build.


Can  not be used to do this?


No. The Javadoc task generates a file then zips it. We need to change 
the timestamp of the file(s) inside the created zip.


??
$ ant javadoc

Dumps everything into output/dist/webapps/docs/, no ZIP files. The 
"javadoc" target executes  tasks and that's it.


What am I missing?


- anyone can build a release from the tag and will get a build that
   includes a signed Windows installer


Oh, that last part is clever, which of course is the whole point of the
detached signatures.


When it is finally working it is going to be really nice. We'll 
effectively have a way for anyone to build from source and confirm that 
they have exactly the same bits as the official release. If provides a 
great way to demonstrate that the RM hasn't tampered with the binaries.


+1

Hopefully, anything Java 11 or later with e.g. "-actlike 1.7" will 
produce binary-identical artifacts. If not, merely stating that a 
particular build was done with some exact JDK version should be enough.


Hopefully we'll be able to find a way to be reasonably flexible on Java 
versions but we'll have to see.


As long as some other party who wishes to confirm the release hasn't 
been tampered-with will be able to use the same toolchain, assuming it's 
documented properly. This is what non-Java projects do for 
repeatability. You can't get the same bytes when using gcc versus clang, 
for example. Same with various versions of those things.


The release-build should probably auto-generate a report of the 
build-chain used to build the release.


Nice idea.

Other potential improvements to the build process I've been 
considering making:


1. $ ant prep-release
Generates a new build.properties file, build-release.properties which 
contains a few useful items:

ant.tstamp.now=[timestamp to be used for repeatable build]
version.suffix= (intentionally blank)
release.asfuser=[username of the user rolling the release]
release.jdk=[details of build environment, perhaps additional items]
release.os=[ibid]

Then modify the build script to include build-release.properties 
before any others and ignore the file if it's missing.


Also this will generate res/maven/build-release.properties which will 
include:

asf.ldap.username
maven.asf.release.deploy.version
gpg.exec

There's no reason these items can't be computed from the outer build 
process and used inside there. It eliminates a few steps for the 
person doing the release.


2. Re-name res/maven/mvn-pub.xml -> build.xml

This saves unnecessary typing during the release. I'll probably 
actually move the file, then create a new one in its place pointing to 
the new one, in case some other downstream process needs that file to 
be in-place.


I doubt that will be necessary. The file is very specific to how the ASF 
is set up.


3. Read build-release.properties in build.xml (nee mvn-pub.xml) as 
well as build.properties and build.properties.default. 
build.properties should be almost unnecessary at this point.


4. Document the mail build s in build.xml (nee mvn-pub.xml)

I'm sure there are other things I've been thinking of. These are the 
ones I have in the front of my mind.


Sounds good. Anything that makes the release process easier is a good 
thing (although compared to what it used to be like the current process 
is a breeze).


:)

Okay, I'll try to do those in advance of the next releases.

-chris

-
To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org
For additional commands, e-mail: dev-h...@tomcat.apache.org

Re: Repeatable builds

[Mark Thomas]

On 23/03/2022 16:11, Christopher Schultz wrote:

Mark,

On 3/22/22 15:30, Mark Thomas wrote:

On 22/03/2022 19:06, Christopher Schultz wrote:

On 3/22/22 13:12, Mark Thomas wrote:




The JSign Ant task that adds the detached signature doesn't close 
the signed file. This causes problems for Ant. I've opened a JSign 
issue [2] for this. I have a locally build version with a hack that 
fixes the issue so I can continue testing.


Do you mean it doesn't call OutputStream.close()? Shouldn't that be 
cleaned up when the process exits?


Effectively, yes. The problem is that the Ant task runs in process so 
the file is still open when the next Ant target tries to work with the 
file. You won't notice the issue running JSign from the command line.


Can we run the jsign task with fork="true" or is there too much 
information in the ant process's memory that would need to be 
insecurely-sent to the jsign process to be prudent?


I don't think it supports it. If it did, that would be another workaround.

No. The Javadoc task generates a file then zips it. We need to change 
the timestamp of the file(s) inside the created zip.


??
$ ant javadoc

Dumps everything into output/dist/webapps/docs/, no ZIP files. The 
"javadoc" target executes  tasks and that's it.


What am I missing?


Look in the individual Javadoc directories. Each one has (I think) 3 zip 
files that contain various indexes.



Hopefully, anything Java 11 or later with e.g. "-actlike 1.7" will 
produce binary-identical artifacts. If not, merely stating that a 
particular build was done with some exact JDK version should be enough.


Hopefully we'll be able to find a way to be reasonably flexible on 
Java versions but we'll have to see.


As long as some other party who wishes to confirm the release hasn't 
been tampered-with will be able to use the same toolchain, assuming it's 
documented properly. This is what non-Java projects do for 
repeatability. You can't get the same bytes when using gcc versus clang, 
for example. Same with various versions of those things.


Same tool-chain should definitely work. But I'd still like to see how 
much flexibility - if any - we have in versions of Java and Ant.


Sounds good. Anything that makes the release process easier is a good 
thing (although compared to what it used to be like the current 
process is a breeze).


:)

Okay, I'll try to do those in advance of the next releases.


Cool.

Mark

-
To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org
For additional commands, e-mail: dev-h...@tomcat.apache.org

Re: Repeatable builds

[Christopher Schultz]

Mark,

On 3/23/22 12:18, Mark Thomas wrote:

On 23/03/2022 16:11, Christopher Schultz wrote:

??
$ ant javadoc

Dumps everything into output/dist/webapps/docs/, no ZIP files. The 
"javadoc" target executes  tasks and that's it.


What am I missing?


Look in the individual Javadoc directories. Each one has (I think) 3 zip 
files that contain various indexes.


$ ant clean javascript
[...]
$ find output/dist/webapps/docs/ -name "*.zip" -o -name "*.jar"
$

I don't see those. Maybe it's my Java version?

$ java -version
openjdk version "16.0.2" 2021-07-20
OpenJDK Runtime Environment (build 16.0.2+7-67)
OpenJDK 64-Bit Server VM (build 16.0.2+7-67, mixed mode, sharing)

$ head output/dist/webapps/docs/servletapi/index.html





Overview (Servlet {servlet.spec.version} API Documentation - 
Apache Tomcat 10.1.0-M13-dev)






href="script-dir/jquery-ui.min.css" title="Style">
title="Style">


src="script-dir/jquery-3.5.1.min.js">




So it's definitely using the Java 16 javadoc engine.

Hopefully, anything Java 11 or later with e.g. "-actlike 1.7" will 
produce binary-identical artifacts. If not, merely stating that a 
particular build was done with some exact JDK version should be enough.


Hopefully we'll be able to find a way to be reasonably flexible on 
Java versions but we'll have to see.


As long as some other party who wishes to confirm the release hasn't 
been tampered-with will be able to use the same toolchain, assuming 
it's documented properly. This is what non-Java projects do for 
repeatability. You can't get the same bytes when using gcc versus 
clang, for example. Same with various versions of those things.


Same tool-chain should definitely work. But I'd still like to see how 
much flexibility - if any - we have in versions of Java and Ant.


Sounds good. Anything that makes the release process easier is a good 
thing (although compared to what it used to be like the current 
process is a breeze).


:)

Okay, I'll try to do those in advance of the next releases.


Cool.


I'm having some trouble getting  to give me 
current-timestamp-in-seconds. I can get the ISO date just fine.


This SO answer [https://stackoverflow.com/a/25535927/276232] doesn't 
seems to work out--of-the-box; I get an error about not being able to 
initialize the javascript engine.


I tried the obvious pattern="sss" but I just got a bunch of 
zeros followed by the seconds-in-the-current-minute. :/


I have it building the following build-release.properties file:

=== CUT ===
# These ant.tstamp properties specify the build-timestamp for repeatable 
builds.

ant.tstamp.now=41
ant.tstamp.now.iso=2022-03-23T16:48:41Z
# This is the ASF user who is the Release Manager for this release.
release.asfusername=schultz
# Set the version-suffix to "" (empty string) as this is not a 
development release.

version.suffix=
# Release build environment information:
release.java.version=16.0.2
release.java.vm.name=OpenJDK 64-Bit Server VM
release.java.vm.version=16.0.2+7-67
release.os=x86_64 Mac OS X 11.5
release.default.file.encoding=UTF-8
=== CUT ===

So... almost everything I want :)

Any ideas for the "time in seconds" thing? (I may be able to get ant 
working with javascript, but it requires a bunch of dependencies 
including GraalVM, etc. - !!!).


-chris

-
To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org
For additional commands, e-mail: dev-h...@tomcat.apache.org