dweiss opened a new pull request, #14764: URL: https://github.com/apache/lucene/pull/14764
> **Short version** (TL;DR;) I don't expect people to go through the entire diff of this patch - it's large. Most files haven't really changed much - they're just in a different location and perhaps have been reformatted (gradle/groovy scripts are now tidied up). **What I would like you to do**, if you have time, is to switch to the PR's branch and try whatever workflow you are accustomed to - if things work for you the same way they did before. I am looking for regressions and perhaps an approval to apply this to the main branch and just fix things as we discover them. I have been working for over two months trying to convert the existing gradle build into a "more modern" gradle build setup. The motivation for this is mostly to increase the speed of build execution but there are also deprecations and looming changes in gradle that will bite us hard if we don't try to clean up. I have been doing this on my own because it's a large effort that requires frequent backtracking... and I also didn't have a clear vision where I wanted it to go. I think it's in a better form now than it is on main so I'd like you to review and consider switching over to this new setup. It isn't all sweet. In fact, it's a bit disheartening. Read on, sorry for the lengthy introduction. I really like the "aspect-oriented" setup of build files, where each file was configuring a certain aspect of the build: for example spotless or forbiddenapis checker applied across all projects. I really wanted to somehow preserve this but also speed up the initialization and evaluation phases. I thought gradle's "convention plugins" would be a good fit because they're true "gradle plugins" (are precompiled, live in a separate project) but allow us to keep the liberal style of build scripts. I set out to convert all those existing build scripts we currently have into convention plugins. Two months later, things generally work well - these scripts are precompiled, the syntax highlighting and suggestions work better, there are fewer headaches with script classpath... but the initialization time nearly _doubled_ instead of going down. This first-run generates some stubs and compiles all convention plugins. Once they're compiled, things seem to be a bit faster (even from a cold start, without a daemon). But the difference isn't huge - maybe one or two seconds (it takes 8 seconds to get to the execution phase on my machine). The first run though (from a clean checkout) is twice longer (24 seconds instead of 12 on my machine)... I don't know why it's so slow. My long-term plan is to move some of those convention plugins to Java, which should be faster to compile and execute (...and read?). This patch is already large and I didn't want it to grow beyond comprehension - most of the code remains identical as in main, it is just moved, reshuffled or slightly modernized (to use build options, lazy evaluation APIs, etc.). Now, the slowdown was a bummer but there are also benefits. I already mentioned a few: these convention plugins live under a separate composite gradle project and use a more reasonable infrastructure (dependencies, cross-visibility to Java classes, etc.). It should make it much easier to gradually improve this over time compared to build scripts. I also really like the replacement for "propertyOrDefault" - a custom plugin that implements "build options" - tweakable build parameters like ```tests.iters``` or ```tests.jvms``` with the difference being that now one can list _all_ the configurable parameters of each Lucene project and see their current values (and the source they come from). It works like a charm on this branch and I can think of a few things to make it even better. Just look at this:  So, please try to switch to the PR's branch and run whatever you typically run. See if it works. Share your thoughts. Below is a more structured record of larger changes included in this patch. ### ```buildOptions``` plugin instead of ```propertyOrDefault``` method All "propertyOrDefault*" methods have been removed and an external plugin for configuring various build options is now used. This is the plugin (blame me for any shortcomings): https://github.com/carrotsearch/gradle-build-infra#plugin-comcarrotsearchgradlebuildinfrabuildoptionsbuildoptionsplugin this plugin will source the value of a given build option from several places, in order: - system property (```-Dfoo=value```), - gradle property (```-Pfoo=value```), - environment variable (```foo=value ./gradlew ...```) - a *non-versioned*, ```build-options.local.properties``` property file (for your personal, local tweaks), - a *versioned* ```build-options.properties``` property file. It works much like before - you can override any build option temporarily by passing, for example, ```-Ptests.verbose=true```. But you can also make such changes locally persistent by placing them in your ```build-options.local.properties``` file. Alternatively, the generated ```gradle.properties``` is also supported since it declares project properties (equivalent to ```-Pkey=value```). At some point in the future, we may want to keep gradle.properties versioned though, so it's probably a good idea to switch over to ```build-options.local.properties```. The biggest gain from using build options is that you can now see all declared options and their values, including their source (where their value comes from). To see what I mean, try running these: ``` ./gradlew :buildOptions ./gradlew -p lucene/core buildOptions ``` ### Lucene 'base version' string moved * Various Lucene's "version" properties are now full build options (as described above). Their default values are in ```build-options.properties```. You can override any of ```version.suffix```, ```version.base``` or ```version.release``` using any of the build option plugin's methods. If you'd like to bump Lucene's base version, it now lives in the versioned ```build-options.properties```. ### Tweaks to properties and options * ```runtime.java.home``` is a build option now but the support for ```RUNTIME_JAVA_HOME``` env. variable has been implemented for backward compatibility. * ```tests.neverUpToDate``` option is renamed ```tests.rerun```. The value of ```true``` indicates test tasks will re-run even if nothing has changed. ### Changes to project structure * ```build-tools/missing-doclet``` is now a regular project within Lucene (not an included composite project). This simplifies the build and ensures we apply the same checks to this subproject as we do to everything else. * all gradle build scripts are converted to convention plugins (in groovy) and live under ```dev-tools/build-infra/src/main/groovy```. The long-term plan is to move some or all of these groovy plugins to Java (so that the code is easier to read and debug for non-groovians). * ```tidy``` is now applied to groovy/gradle scripts (formatting is enforced). ### Security manager support * parts of the build supporting security manager settings have been just removed, without any replacement. ### Other notable changes * The legacy ```precommit``` task has been removed; use gradle's ```check```. * Removed dependency on jgit entirely. This is replaced by forking the system's git in porcelain mode (which should be stable and portable). This logic is implemented in [this plugin](https://github.com/carrotsearch/gradle-build-infra/?tab=readme-ov-file#plugin-comcarrotsearchgradlebuildinfraenvironmentgitinfoplugin). An additional benefit is that all features of git should now work (including worktrees). * Added support for owasp API keys in the form of ```validation.owasp.apikey``` build option. Owasp check is still very, very slow. We should probably just drop it. * I've changed the default on ```gradle.ge``` (Gradle Enterprise, develocity) to ```false``` on non-CI builds. * I've tried to clean up lots and lots of gradle/groovy code related to eager initialization of tasks as well as update deprecated APIs. This isn't always straightforward and I might have broken some things... ### Fixes to existing issues * ```gradlew clean check``` will work now (https://github.com/apache/lucene/issues/13567) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
