[
https://issues.apache.org/jira/browse/LUCENE-9077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dawid Weiss updated LUCENE-9077:
--------------------------------
Description:
This task focuses on providing gradle-based build equivalent for Lucene and
Solr (on master branch). See notes below on why this respin is needed.
The code lives on *gradle-master* branch. It is kept with sync with *master*.
Try running the following to see an overview of helper guides concerning
typical workflow, testing and ant-migration helpers:
gradlew :help
A list of items that needs to be added or requires work. If you'd like to work
on any of these, please add your name to the list. Once you have a patch/ pull
request let me (dweiss) know - I'll try to coordinate the merges.
* (/) Apply forbiddenAPIs
* (/) Generate hardware-aware gradle defaults for parallelism (count of
workers and test JVMs).
* (/) Fail the build if --tests filter is applied and no tests execute during
the entire build (this allows for an empty set of filtered tests at single
project level).
* (/) Port other settings and randomizations from common-build.xml
* (/) Configure security policy/ sandboxing for tests.
* (/) test's console output on -Ptests.verbose=true
* (/) add a :helpDeps explanation to how the dependency system works (palantir
plugin, lockfile) and how to retrieve structured information about current
dependencies of a given module (in a tree-like output).
* (/) jar checksums, jar checksum computation and validation. This should be
done without intermediate folders (directly on dependency sets).
* (/) verify min. JVM version and exact gradle version on build startup to
minimize odd build side-effects
* [DW, in progress] identify and list precommit tasks so that they can be
ported one by one. (Mark's branch has some of this stuff already implemented)
* Repro-line for failed tests/ runs.
Hard-to-implement stuff already investigated:
* (!) *Printing console output of failed tests.* There doesn't seem to be any
way to do this in a reasonably efficient way. There are onOutput listeners but
they're slow to operate and solr tests emit *tons* of output so it's an
overkill.
* (!) *Tests working with security-debug logs or other JVM-early log output*.
From what I can see in the code Gradle's test runner works by redirecting
Java's stdout/ syserr so this just won't work. Perhaps we can spin the
ant-based test runner for such corner-cases.
Of lesser importance:
* add rendering of javadocs (gradlew javadoc) and attaching them to maven
publications.
* Add test 'beasting' (rerunning the same suite multiple times). I'm afraid
it'll be difficult to run it sensibly because gradle doesn't offer cwd
separation for the forked test runners.
* if you diff solr packaged distribution against ant-created distribution
there are minor differences in library versions and some JARs are excluded/
moved around. I didn't try to force these as everything seems to work (tests,
etc.) – perhaps these differences should be fixed in the ant build instead.
* identify and port any other "check" utilities that may be called from ant.
(Mark's branch has some of this stuff already implemented)
* [EOE] identify and port various "regenerate" tasks from ant builds (javacc,
precompiled automata, etc.)
* fill in POM details in gradle/defaults-maven.gradle so that they reflect the
previous content better (dependencies aside).
* Add any IDE integration layers that should be added (I use IntelliJ and it
imports the project out of the box, without the need for any special tuning).
* *Clean up dependencies, especially for Solr*: any \{ transitive = false }
should just explicitly exclude whatever they don't need (and their dependencies
currently declared explicitly should be folded). Figure out which scope to
import a dependency to.
* Add Solr packaging for docs/* (see TODO in packaging/build.gradle; currently
XSLT...)
* I didn't bother adding Solr dist/test-framework to packaging (who'd use it
from a binary distribution?
*{color:#ff0000}Note:{color}* this builds on the work done by Mark Miller and
Cao Mạnh Đạt but also applies lessons learned from those two efforts:
* *Do not try to do too many things at once*. If we deviate too far from
master, the branch will be hard to merge.
* *Do everything in baby-steps* and add small, independent build fragments
replacing the old ant infrastructure.
* *Try to engage people to run, test and contribute early*. It can't be a
one-man effort. The more people understand and can contribute to the build, the
more healthy it will be.
was:
This task focuses on providing gradle-based build equivalent for Lucene and
Solr (on master branch). See notes below on why this respin is needed.
The code lives on *gradle-master* branch. It is kept with sync with *master*.
Try running the following to see an overview of helper guides concerning
typical workflow, testing and ant-migration helpers:
gradlew :help
A list of items that needs to be added or requires work. If you'd like to work
on any of these, please add your name to the list. Once you have a patch/ pull
request let me (dweiss) know - I'll try to coordinate the merges.
* (/) Apply forbiddenAPIs
* (/) Generate hardware-aware gradle defaults for parallelism (count of
workers and test JVMs).
* (/) Fail the build if --tests filter is applied and no tests execute during
the entire build (this allows for an empty set of filtered tests at single
project level).
* (/) Port other settings and randomizations from common-build.xml
* (/) Configure security policy/ sandboxing for tests.
* (/) test's console output on -Ptests.verbose=true
* (/) add a :helpDeps explanation to how the dependency system works (palantir
plugin, lockfile) and how to retrieve structured information about current
dependencies of a given module (in a tree-like output).
* (/) jar checksums, jar checksum computation and validation. This should be
done without intermediate folders (directly on dependency sets).
* (/) verify min. JVM version and exact gradle version on build startup to
minimize odd build side-effects
* [DW, in progress] identify and list precommit tasks so that they can be
ported one by one. (Mark's branch has some of this stuff already implemented)
* Repro-line for failed tests/ runs.
Hard-to-implement stuff already investigated:
* (!) *Printing console output of failed tests.* There doesn't seem to be any
way to do this in a reasonably efficient way. There are onOutput listeners but
they're slow to operate and solr tests emit *tons* of output so it's an
overkill. The only solution I think would be feasible is to parse test report
XMLs after the tests are complete and collect the output of failed tests from
there. The downside is that XMLs contain separated sysout/syserr.
* (!) *Tests working with security-debug logs or other JVM-early log output*.
From what I can see in the code Gradle's test runner works by redirecting
Java's stdout/ syserr so this just won't work. Perhaps we can spin the
ant-based test runner for such corner-cases.
Of lesser importance:
* add rendering of javadocs (gradlew javadoc) and attaching them to maven
publications.
* Add test 'beasting' (rerunning the same suite multiple times). I'm afraid
it'll be difficult to run it sensibly because gradle doesn't offer cwd
separation for the forked test runners.
* if you diff solr packaged distribution against ant-created distribution
there are minor differences in library versions and some JARs are excluded/
moved around. I didn't try to force these as everything seems to work (tests,
etc.) – perhaps these differences should be fixed in the ant build instead.
* identify and port any other "check" utilities that may be called from ant.
(Mark's branch has some of this stuff already implemented)
* [EOE] identify and port various "regenerate" tasks from ant builds (javacc,
precompiled automata, etc.)
* fill in POM details in gradle/defaults-maven.gradle so that they reflect the
previous content better (dependencies aside).
* Add any IDE integration layers that should be added (I use IntelliJ and it
imports the project out of the box, without the need for any special tuning).
* *Clean up dependencies, especially for Solr*: any \{ transitive = false }
should just explicitly exclude whatever they don't need (and their dependencies
currently declared explicitly should be folded). Figure out which scope to
import a dependency to.
* Add Solr packaging for docs/* (see TODO in packaging/build.gradle; currently
XSLT...)
* I didn't bother adding Solr dist/test-framework to packaging (who'd use it
from a binary distribution?
*{color:#ff0000}Note:{color}* this builds on the work done by Mark Miller and
Cao Mạnh Đạt but also applies lessons learned from those two efforts:
* *Do not try to do too many things at once*. If we deviate too far from
master, the branch will be hard to merge.
* *Do everything in baby-steps* and add small, independent build fragments
replacing the old ant infrastructure.
* *Try to engage people to run, test and contribute early*. It can't be a
one-man effort. The more people understand and can contribute to the build, the
more healthy it will be.
> Gradle build
> ------------
>
> Key: LUCENE-9077
> URL: https://issues.apache.org/jira/browse/LUCENE-9077
> Project: Lucene - Core
> Issue Type: Task
> Reporter: Dawid Weiss
> Assignee: Dawid Weiss
> Priority: Major
> Fix For: master (9.0)
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> This task focuses on providing gradle-based build equivalent for Lucene and
> Solr (on master branch). See notes below on why this respin is needed.
> The code lives on *gradle-master* branch. It is kept with sync with *master*.
> Try running the following to see an overview of helper guides concerning
> typical workflow, testing and ant-migration helpers:
> gradlew :help
> A list of items that needs to be added or requires work. If you'd like to
> work on any of these, please add your name to the list. Once you have a
> patch/ pull request let me (dweiss) know - I'll try to coordinate the merges.
> * (/) Apply forbiddenAPIs
> * (/) Generate hardware-aware gradle defaults for parallelism (count of
> workers and test JVMs).
> * (/) Fail the build if --tests filter is applied and no tests execute
> during the entire build (this allows for an empty set of filtered tests at
> single project level).
> * (/) Port other settings and randomizations from common-build.xml
> * (/) Configure security policy/ sandboxing for tests.
> * (/) test's console output on -Ptests.verbose=true
> * (/) add a :helpDeps explanation to how the dependency system works
> (palantir plugin, lockfile) and how to retrieve structured information about
> current dependencies of a given module (in a tree-like output).
> * (/) jar checksums, jar checksum computation and validation. This should be
> done without intermediate folders (directly on dependency sets).
> * (/) verify min. JVM version and exact gradle version on build startup to
> minimize odd build side-effects
> * [DW, in progress] identify and list precommit tasks so that they can be
> ported one by one. (Mark's branch has some of this stuff already implemented)
> * Repro-line for failed tests/ runs.
> Hard-to-implement stuff already investigated:
> * (!) *Printing console output of failed tests.* There doesn't seem to be
> any way to do this in a reasonably efficient way. There are onOutput
> listeners but they're slow to operate and solr tests emit *tons* of output so
> it's an overkill.
> * (!) *Tests working with security-debug logs or other JVM-early log
> output*. From what I can see in the code Gradle's test runner works by
> redirecting Java's stdout/ syserr so this just won't work. Perhaps we can
> spin the ant-based test runner for such corner-cases.
> Of lesser importance:
> * add rendering of javadocs (gradlew javadoc) and attaching them to maven
> publications.
> * Add test 'beasting' (rerunning the same suite multiple times). I'm afraid
> it'll be difficult to run it sensibly because gradle doesn't offer cwd
> separation for the forked test runners.
> * if you diff solr packaged distribution against ant-created distribution
> there are minor differences in library versions and some JARs are excluded/
> moved around. I didn't try to force these as everything seems to work (tests,
> etc.) – perhaps these differences should be fixed in the ant build instead.
> * identify and port any other "check" utilities that may be called from ant.
> (Mark's branch has some of this stuff already implemented)
> * [EOE] identify and port various "regenerate" tasks from ant builds
> (javacc, precompiled automata, etc.)
> * fill in POM details in gradle/defaults-maven.gradle so that they reflect
> the previous content better (dependencies aside).
> * Add any IDE integration layers that should be added (I use IntelliJ and it
> imports the project out of the box, without the need for any special tuning).
> * *Clean up dependencies, especially for Solr*: any \{ transitive = false }
> should just explicitly exclude whatever they don't need (and their
> dependencies currently declared explicitly should be folded). Figure out
> which scope to import a dependency to.
> * Add Solr packaging for docs/* (see TODO in packaging/build.gradle;
> currently XSLT...)
> * I didn't bother adding Solr dist/test-framework to packaging (who'd use it
> from a binary distribution?
>
> *{color:#ff0000}Note:{color}* this builds on the work done by Mark Miller and
> Cao Mạnh Đạt but also applies lessons learned from those two efforts:
> * *Do not try to do too many things at once*. If we deviate too far from
> master, the branch will be hard to merge.
> * *Do everything in baby-steps* and add small, independent build fragments
> replacing the old ant infrastructure.
> * *Try to engage people to run, test and contribute early*. It can't be a
> one-man effort. The more people understand and can contribute to the build,
> the more healthy it will be.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
