> > perhaps adding a paragraph or a Q/A on > > https://maven.apache.org/guides/mini/ > > guide-reproducible-builds.html > > <https://maven.apache.org/guides/mini/guide-reproducible-builds.html> > > would be the best strategy, to describe the > > different ways of configuring timestamps in archives, depending on usage > > If the how to release with that constraint is solved I agree it is just > about documenting it. it is currently solved on releases with maven-release-plugin MRELEASE-1029 IMHO, we can add explanations for 2 use cases: 1. people wanting to have changing timestamps for SNAPSHOTs 2. people not using maven-release-plugin to produce their releases here, using last git commit timestamp seems a good approach
WDYT? > > ok, I understand about checking fixed timestamp in archive, not really > > what you > > call "non portable params". > > And sorry, but even checking fixed timestamps won't always work: in the > > case of > > maven-shade-plugin, the algorithm used when shading from libraries was not > > to > > override timestamp but keep original > > exactly but then the flow is ready and it is just about plugins which are > not reproducible plugins friendly. -1 m-shade-p is reproducible: it's just that reproducible does not mean fixed timestamp for absolutely everything, but just reproducible > Concretely, once you solved the portability (eol on linux/win) and > timestamp issue, if the plugins respect that (can be a transformer wrapper > for shade for ex) then all the needed info are there and only the plugin > must respect the info. as documented, OS portability is out of scope: OS (or more precisely LF vs CRLF) is part of the environment required to rebuild, like the major JDK version FYI, we have workaround for rebuilding on Linux a release done on Windows: the only part where I need help is how to get sources from Git with CRLF. Currently, I have to do the rebuild from -source-release.zip Help appreciated on https://github.com/jvm-repo-rebuild/reproducible-central/issues/9 > If release plugin does not put the info in the pom then all plugins have to > do it and have their own standard config which is a wider pain IMHO. when people use maven-release-plugin, everything is done for releases (whatever the scm used, or source tarballs disconnected from scm): didn't you try? when people choose not to use maven-release-plugin, they to choose to live the hard way, that's their choice :) the only case where people using maven-release-plugin are not covered is the case of SNAPSHOTs when they need to have a moving timestamp = the origin of our discussion Le dimanche 19 avril 2020, 11:19:50 CEST Romain Manni-Bucau a écrit : > Le dim. 19 avr. 2020 à 10:36, Hervé BOUTEMY <[email protected]> a > > écrit : > > Le samedi 18 avril 2020, 22:59:42 CEST Romain Manni-Bucau a écrit : > > > > If you tell me every servlet container uses timestamp of entries in > > > > wars > > > > > > to > > > > change their behaviour, I'm ok and will stop talking about Tomcat > > > > only, > > > > but > > > > more about "servlet containers" (or better if you propose another > > > > term): I > > > > > > can > > > > understand how this is useful for client HTTP cache of static content. > > > > If > > > > > > there is also such handling in a servlet container for classes in war, > > > > or > > > > > > jars > > > > in wars, I'm eager to learn (and learn for jars in wars if it's only > > > > the > > > > > > .class timestamp in jars that is used, or also the timestamp of the > > > > jars > > > > > > in > > > > the war) > > > > > > Yes all do, even java -jar mywebapp.jar. > > > I also know several batches or standalone apps using that to log the > > > deployed version (very useful in prod where date helps more than numbers > > > for CD). > > > > ok > > perhaps adding a paragraph or a Q/A on > > https://maven.apache.org/guides/mini/ > > guide-reproducible-builds.html > > <https://maven.apache.org/guides/mini/guide-reproducible-builds.html> > > would be the best strategy, to describe the > > different ways of configuring timestamps in archives, depending on usage > > If the how to release with that constraint is solved I agree it is just > about documenting it. > > > > > > Now, technically, a new phase in release plugin adding the timestamp > > > > > (and > > > > > eol?) in the pom in prepare phase works I think and is (almost) > > > > portable > > > > > > > for the jar, war and ear packagings for users, no? I can envision an > > > > > <enforceReproducible> in release plugin maybe. > > > > > > > > > > Am I missing something? > > > > > > > > I just don't understand how what you describe is different from what > > > > was > > > > > > done > > > > in maven-release-plugin in MRELEASE-1029 [1] > > > > And I don't understand what "enforceReproducible" can mean in release > > > > plugin > > > > (enforcing is really hard, believe me: I still don't fully understand > > > > maven- > > > > site-plugin 3.9.0 release was not reproducible, because of > > > > maven-invoker- > > > > > > plugin interaction during reference release build) > > > > > > Enforcing the timestamp and not portable params helps. Must be hardcoded > > > > in > > > > > the pom of the tag imo otherwise it wouldnt exist and be really > > > reproduc*a*ble. > > > > ok, I understand about checking fixed timestamp in archive, not really > > what you > > call "non portable params". > > And sorry, but even checking fixed timestamps won't always work: in the > > case of > > maven-shade-plugin, the algorithm used when shading from libraries was not > > to > > override timestamp but keep original > > exactly but then the flow is ready and it is just about plugins which are > not reproducible plugins friendly. > Concretely, once you solved the portability (eol on linux/win) and > timestamp issue, if the plugins respect that (can be a transformer wrapper > for shade for ex) then all the needed info are there and only the plugin > must respect the info. > If release plugin does not put the info in the pom then all plugins have to > do it and have their own standard config which is a wider pain IMHO. > > > > > thanks for the interest and strong feedback: it's hard but useful > > > > > > > > Regards, > > > > > > > > Hervé > > > > > > > > [1] > > > > https://maven.apache.org/guides/mini/guide-reproducible-builds.html > > > > > > Le samedi 18 avril 2020, 19:21:07 CEST Romain Manni-Bucau a écrit : > > > > > Hervé, can you clarify why you mention so strongly tomcat? > > > > > > > > > > Just out of my head tomcat, undertow, vertx, netty, camel, all cxf > > > > > transports (including standalone, talend sdk, wildfly, tomee, > > > > > meecrowave, > > > > > restlet, quarkus, helidon, jetty, resin, play, akka-http, xnio and > > > > much > > > > > > > more are affected at war and jar levels so why would tomcat be > > > > specific? > > > > > > > It is just impacting any application using timestamps of what is > > > > > > > > packaged. > > > > > > > > > To rephrase it: all impl are quite equivalent. > > > > > > > > > > Now, technically, a new phase in release plugin adding the timestamp > > > > > (and > > > > > eol?) in the pom in prepare phase works I think and is (almost) > > > > portable > > > > > > > for the jar, war and ear packagings for users, no? I can envision an > > > > > <enforceReproducible> in release plugin maybe. > > > > > > > > > > Am I missing something? > > > > > > > > > > Le sam. 18 avr. 2020 à 15:44, Hervé BOUTEMY <[email protected]> > > > > a > > > > > > > écrit : > > > > > > SNAPSHOTs and releases are similar from certain points of view and > > > > > > different > > > > > > from others: what interests me is that we vote on releases, which > > > > are > > > > > > the > > > > > > > > > > only > > > > > > official states we publish (as a source-release.zip tarball > > > > > > > > disconnected > > > > > > > > > > from > > > > > > scm), but we don't do it on SNAPSHOTs, which are only intermediate > > > > > > > > states > > > > > > > > > > we > > > > > > only keep on scm. > > > > > > > > > > > > But I understand that from a Tomcat perspective, hosting SNAPSHOTs > > > > or > > > > > > > > releases > > > > > > is quite similar. > > > > > > > > > > > > Perhaps this question of Reproducible Builds and the impact of > > > > fixed > > > > > > > > timestamp > > > > > > in jar/wars would be something that would require some > > > > Tomcat-specific > > > > > > > > documentation (that perhaps other servlet containers don't > > > > implement > > > > > > the > > > > > > > > > > same > > > > > > way), that we could point to in maven-war-plugin? > > > > > > > > > > > > We should probably switch this discussion to Tomcat ML, users or > > > > dev. > > > > > > > > Regards, > > > > > > > > > > > > Hervé > > > > > > > > > > > > Le samedi 18 avril 2020, 12:34:01 CEST Romain Manni-Bucau a écrit : > > > > > > > Snapshot or releases are not different until you force a > > > > pre-release > > > > > > > > > step > > > > > > > to hardcode the timestamp in the pom which would defeat release > > > > > > > > process > > > > > > > > > > > until done in release plugin IMHO. > > > > > > > Using scm meta is not that bad in that aspect. > > > > > > > > > > > > > > Side note: this is not a war plugin issue and affects jar plugin > > > > too > > > > > > > > cause > > > > > > > > > > > > > they regularly host we resources too (through standard servlet > > > > > > > > layout or > > > > > > > > > > > not). > > > > > > > > > > > > > > So overall sounds like a transversal archiver fix and not by > > > > type to > > > > > > me. > > > > > > > > > > > Le sam. 18 avr. 2020 à 12:14, Hervé BOUTEMY < > > > > [email protected]> > > > > > > a > > > > > > > > > > > écrit : > > > > > > > > keeping SNAPSHOTs reproducible with calculated value for > > > > timestamp > > > > > > is > > > > > > > > > > an > > > > > > > > > > > > > > option that can be chosen by some people > > > > > > > > assuming SNAPSHOTs are not reproducible seems a reasonable > > > > option: > > > > > > > > requirement > > > > > > > > for reproducible SNAPSHOTs is really different from > > > > > > > > requirement > > > > > > > > for > > > > > > > > releases > > > > > > > > > > > > > > > > IMHO, the misc strategies available for war files, against the > > > > way > > > > > > > > > > some > > > > > > > > web > > > > > > > > servers use timestamp, will require dedicated documentation in > > > > > > > > > > > > maven-war- > > > > > > > > > > > > > > plugin > > > > > > > > > > > > > > > > and one key war-specific feature: disable reproducible output > > > > for > > > > > > > > > > SNAPSHOTs, > > > > > > > > which is one strategy that can be chosen (Jira issue yet to be > > > > > > > > created) > > > > > > > > > > > > > > > > Will require help from others to write this doc before the > > > > > > > > maven-war-plugin > > > > > > > > release. > > > > > > > > > > > > > > > > Regards, > > > > > > > > > > > > > > > > Hervé > > > > > > > > > > > > > > > > Le samedi 18 avril 2020, 10:30:21 CEST Romain Manni-Bucau a > > > > écrit > > > > > > > > > > > Hi everyone, > > > > > > > > > > > > > > > > > > I got the same issue for my current work webapps (not at > > > > archive > > > > > > > > level > > > > > > > > > > > > > > but > > > > > > > > > > > > > > > > > docker image level - I'm using jib and it enforces a > > > > "constant" > > > > > > > > > > timestamp). > > > > > > > > > > > > > > > > > I solved it by using last commit timestamp as file > > > > > > > > > timestamp. > > > > > > > > Indeed > > > > > > > > > > it > > > > > > > > > > > > > > can > > > > > > > > > > > > > > > > > miss a "strict" cache hit but globally it is a good > > > > compromise > > > > > > and > > > > > > > > > > guess > > > > > > > > > > > > > > it > > > > > > > > > > > > > > > > > can be reused for reproducible builds. > > > > > > > > > In any case, a project using a scm will be reproducible only > > > > > > > > > > > > regarding a > > > > > > > > > > > > > > > commit so it sounds like the least bad compromise. > > > > > > > > > wdyt? > > > > > > > > > > > > > > > > > > Romain Manni-Bucau > > > > > > > > > @rmannibucau <https://twitter.com/rmannibucau> | Blog > > > > > > > > > <https://rmannibucau.metawerx.net/> | Old Blog > > > > > > > > > <http://rmannibucau.wordpress.com> | Github < > > > > > > > > > > > > > > > > https://github.com/rmannibucau> > > > > > > > > > > > > > > > > > | LinkedIn <https://www.linkedin.com/in/rmannibucau> | Book > > > > > > > > > > > > > > > > > > < > > > > https://www.packtpub.com/application-development/java-ee-8-high-performanc > > > > > > > > > > e > > > > > > > > > > > > > > > > > Le sam. 18 avr. 2020 à 10:08, Hervé BOUTEMY < > > > > > > > > [email protected]> > > > > > > > > > > a > > > > > > > > > > > > > > > écrit : > > > > > > > > > > Le vendredi 17 avril 2020, 22:20:11 CEST Michael Osipov a > > > > > > > > écrit : > > > > > > > > > > > Am 2020-04-17 um 22:00 schrieb [email protected]: > > > > > > > > > > > > Reproducible Builds is now implemented in many plugins: > > > > it's > > > > > > > > > > time > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > > work > > > > > > > > > > > > > > > > > > > > > > on reproducible war files. > > > > > > > > > > > > > > > > > > > > > > > > I created MWAR-432 issue and implemented classical > > > > > > > > > > > > Reproducible > > > > > > > > > > > > jar > > > > > > > > > > > > > > > > > > > > output > > > > > > > > > > > > > > > > > > > > > > in corresponding branch. > > > > > > > > > > > > > > > > > > > > > > > > But in our discussion in november [1], an issue was > > > > > > > > reported > > > > > > > > > > for > > > > > > > > > > > > > > > > unchanged > > > > > > > > > > > > > > > > > > > > > > timestamp in SNAPSHOTs war regarding Tomcat detection > > > > > > > > > > > > algorithm of > > > > > > > > > > > > > > > > > > changed content. > > > > > > > > > > > > > > > > > > > > > > If you are referring to Tomcat's ETag calculation, it > > > > uses > > > > > > both > > > > > > > > > > > > > > > timestamp and file size. The weak ETag will change as > > > > soon > > > > > > > > > > > > > as > > > > > > > > > > > the > > > > > > > > > > > > > > > > fize > > > > > > > > > > > > > > > > > > > size changes. Of course, this is a problem because if > > > > > > > > > > > the > > > > > > > > file > > > > > > > > > > > > changes, > > > > > > > > > > > > > > > > > > > but the content length does not, the ETag won't. > > > > > > > > > > > > > > > > > > > > > > From a Tomcat perspective this does not matter because > > > > > > > > > > > every > > > > > > > > > > > > > > > > deployment > > > > > > > > > > > > > > > > > > > has its own class loader and in-memory cache. > > > > > > > > > > > > > > > > > > > > > > From a client's one, yes. The client will receive a 304 > > > > and > > > > > > > > > > > > > read > > > > > > > > > > > > > > > > from > > > > > > > > > > > > > > > > > > > cache although the file has been changed. > > > > > > > > > > > > > > > > > > > > > > > Should we just disable reproducible wars for SNAPSHOTs > > > > and > > > > > > > > enable > > > > > > > > > > > > > > it > > > > > > > > > > > > > > > > > > only > > > > > > > > > > > > > > > > > > > > > > for releases? Should we add a boolean option for > > > > people to > > > > > > > > decide > > > > > > > > > > > > > > > > whether > > > > > > > > > > > > > > > > > > > > > > they want reproducible SNAPSHOTs? > > > > > > > > > > > > > > > > > > > > > > It should be a user choice. > > > > > > > > > > > > > > > > > > > > > > > Is there any idea on how we could manage output > > > > timestamp > > > > > > for > > > > > > > > > > > > > > SNAPSHOTs? > > > > > > > > > > > > > > > > > > > > > You could do baselining: > > > > > > > > > > > > > > > > > > > > > > * All files which have been changed before output > > > > timestamp, > > > > > > > > > > > > > will > > > > > > > > > > > > > > > > have > > > > > > > > > > > > > > > > > > > output timestamp > > > > > > > > > > > * All other files will retain their timestamp. > > > > > > > > > > > > > > > > > > > > > > A changed file will be immediately reflected by its new > > > > > > > > > > > > timestamp. > > > > > > > > > > > > > > > > that means "not reproducible", then if we're at that > > > > > > > > > > level, > > > > > > > > let's > > > > > > > > > > just > > > > > > > > > > > > > > not > > > > > > > > > > > > > > > > > > apply any timestamp calculations: it's just disabling > > > > > > > > reproducible > > > > > > > > > > > > output > > > > > > > > > > > > > > > > > > for > > > > > > > > > > SNASPHOTs > > > > > > > > > > > > > > > > > > > > > Please also ask on [email protected], I might have missed > > > > > > > > other > > > > > > > > > > use > > > > > > > > > > > > > > > > cases. > > > > > > > > > > good idea, thank you > > > > > > > > > > > > > > > > > > > > > Michael > > > > -------------------------------------------------------------------- > > > > > > > > > > > > > - > > > > > > > > > > > To unsubscribe, e-mail: [email protected] > > > > > > > > > > > > > For additional commands, e-mail: > > [email protected] > > > > > > --------------------------------------------------------------------- > > > > > > > > > > > > To unsubscribe, e-mail: [email protected] > > > > > > > > > > For additional commands, e-mail: [email protected] > > > > > > > > --------------------------------------------------------------------- > > > > > > > > > > > > To unsubscribe, e-mail: [email protected] > > > > > > > > For additional commands, e-mail: [email protected] > > > > --------------------------------------------------------------------- > > > > > > > > To unsubscribe, e-mail: [email protected] > > > > > > For additional commands, e-mail: [email protected] > > > > > > > > --------------------------------------------------------------------- > > > > To unsubscribe, e-mail: [email protected] > > > > For additional commands, e-mail: [email protected] > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: [email protected] > > For additional commands, e-mail: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
