[jira] [Commented] (LUCENE-9809) Adapt Release Wizard to only release Lucene
[ https://issues.apache.org/jira/browse/LUCENE-9809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17433409#comment-17433409 ] Dawid Weiss commented on LUCENE-9809: - > There does not seem to be any configuration for specifying the folder where > the signed and voted upon artifacts reside. They are not materialized anywhere locally (well, they are - in each module's build folder). They are staged directly from their respective build locations. You shouldn't try to work around that - the right way to centralize them is in the distribution module: assemble maven artifacts in a local folder "repository". As for revealing the password - I think if you pass it via environment variables the logs won't contain the password anywhere. I'm not 100% sure because, well, I didn't write gradle... > Adapt Release Wizard to only release Lucene > --- > > Key: LUCENE-9809 > URL: https://issues.apache.org/jira/browse/LUCENE-9809 > Project: Lucene - Core > Issue Type: Sub-task >Reporter: Jan Høydahl >Assignee: Jan Høydahl >Priority: Major > Fix For: main (9.0) > > Time Spent: 1h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (LUCENE-10200) Restructure and modernize the release artifacts
[ https://issues.apache.org/jira/browse/LUCENE-10200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dawid Weiss updated LUCENE-10200: - Description: This is an umbrella issue for various sub-tasks as per my e-mail [1]. [1] https://markmail.org/thread/f7yrggnynq2ijgmy In this order, perhaps: * Apply small text file changes (LUCENE-10163) * Simplify artifacts (LUCENE-10199 drop ZIP binary), (LUCENE-10192 drop third party JARs). * Create an additional binary artifact for Luke (LUCENE-9978). * Review the content of licenses/ - there are some entries there that relate to tests only (jetty) or oddballs like elegant-icon-font. * Test everything with the smoke tester. was: This is an umbrella issue for various sub-tasks as per my e-mail [1]. [1] https://markmail.org/thread/f7yrggnynq2ijgmy In this order, perhaps: * Apply small text file changes (LUCENE-10163) * Simplify artifacts (LUCENE-10199 drop ZIP binary), (LUCENE-10192 drop third party JARs). * Create an additional binary artifact for Luke (LUCENE-9978). * Test everything with the smoke tester. > Restructure and modernize the release artifacts > --- > > Key: LUCENE-10200 > URL: https://issues.apache.org/jira/browse/LUCENE-10200 > Project: Lucene - Core > Issue Type: Task >Reporter: Dawid Weiss >Assignee: Dawid Weiss >Priority: Major > > This is an umbrella issue for various sub-tasks as per my e-mail [1]. > [1] https://markmail.org/thread/f7yrggnynq2ijgmy > In this order, perhaps: > * Apply small text file changes (LUCENE-10163) > * Simplify artifacts (LUCENE-10199 drop ZIP binary), (LUCENE-10192 drop third > party JARs). > * Create an additional binary artifact for Luke (LUCENE-9978). > * Review the content of licenses/ - there are some entries there that relate > to tests only (jetty) or oddballs like elegant-icon-font. > * Test everything with the smoke tester. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (LUCENE-10200) Restructure and modernize the release artifacts
[ https://issues.apache.org/jira/browse/LUCENE-10200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dawid Weiss updated LUCENE-10200: - Description: This is an umbrella issue for various sub-tasks as per my e-mail [1]. [1] https://markmail.org/thread/f7yrggnynq2ijgmy In this order, perhaps: * Apply small text file changes (LUCENE-10163) * Simplify artifacts (LUCENE-10199 drop ZIP binary), (LUCENE-10192 drop third party JARs). * Create an additional binary artifact for Luke (LUCENE-9978). * Review the content of licenses/ - there are some entries there that relate to tests only (jetty) or oddballs like elegant-icon-font or a stray pddl*.txt. * Test everything with the smoke tester. was: This is an umbrella issue for various sub-tasks as per my e-mail [1]. [1] https://markmail.org/thread/f7yrggnynq2ijgmy In this order, perhaps: * Apply small text file changes (LUCENE-10163) * Simplify artifacts (LUCENE-10199 drop ZIP binary), (LUCENE-10192 drop third party JARs). * Create an additional binary artifact for Luke (LUCENE-9978). * Review the content of licenses/ - there are some entries there that relate to tests only (jetty) or oddballs like elegant-icon-font. * Test everything with the smoke tester. > Restructure and modernize the release artifacts > --- > > Key: LUCENE-10200 > URL: https://issues.apache.org/jira/browse/LUCENE-10200 > Project: Lucene - Core > Issue Type: Task >Reporter: Dawid Weiss >Assignee: Dawid Weiss >Priority: Major > > This is an umbrella issue for various sub-tasks as per my e-mail [1]. > [1] https://markmail.org/thread/f7yrggnynq2ijgmy > In this order, perhaps: > * Apply small text file changes (LUCENE-10163) > * Simplify artifacts (LUCENE-10199 drop ZIP binary), (LUCENE-10192 drop third > party JARs). > * Create an additional binary artifact for Luke (LUCENE-9978). > * Review the content of licenses/ - there are some entries there that relate > to tests only (jetty) or oddballs like elegant-icon-font or a stray pddl*.txt. > * Test everything with the smoke tester. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-10192) Drop third-party JARs from the binary distribution
[
https://issues.apache.org/jira/browse/LUCENE-10192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17433423#comment-17433423
]
Dawid Weiss commented on LUCENE-10192:
--
I've pushed a commit that makes Luke a part of the binary distribution. This
kind of makes sense to me now - we don't need all of the third-party jars but
we can make the demos and examples (and Luke) functional as a demonstration and
validation of the reason why the binary release exists. The folder structure
looks like this:
{code}
bin
CHANGES.txt
docs
JRE_VERSION_MIGRATION.md
LICENSE.txt
licenses
MIGRATE.md
modules
modules-test-framework
modules-thirdparty
NOTICE.txt
README.md
SYSTEM_REQUIREMENTS.md
{code}
Where {{modules}} contains Lucene JARs (the test framework is separate because
it validates the module system constraints) and {{modules-thirdparty}} contains
all modules required to run Luke (at the moment). This layout simplifies Luke
utility launch scripts greatly, basically to this:
{code}
java --module-path modules;modules-thirdparty -cp
modules-thirdparty/log4j-api-2.13.2.jar --module lucene.luke
{code}
The classpath element is needed because all the modules currently live in the
automatic-name space while log4j-api is a proper module. This can be cleaned up
as a follow-up by making luke a true module (declare its dependencies).
The side-effect of this layout is also that, in the future, you could inspect
Lucene JARs using the module system's --describe-module option. And like I
mentioned, you can run the demo tools in a much more convenient way:
{code}
java --module-path jars --module lucene.demo/org.apache.lucene.demo.IndexFiles
{code}
I think those "demos" could also benefit from a rewrite (use multiple cores to
index)... but it's an unrelated issue. If we decide it's the direction to
follow then it'd also make LUCENE-9978 invalid (since the binary distribution
would contain a fully-functional Luke as a first-class citizen and user of the
binary modules).
> Drop third-party JARs from the binary distribution
> --
>
> Key: LUCENE-10192
> URL: https://issues.apache.org/jira/browse/LUCENE-10192
> Project: Lucene - Core
> Issue Type: Task
>Reporter: Dawid Weiss
>Assignee: Dawid Weiss
>Priority: Trivial
> Time Spent: 10m
> Remaining Estimate: 0h
>
> [~janhoy] Are we ready (with respect to scripts) for this change? I'd like to
> do it but I'm not sure whether the release wizard doesn't depend on it
> somehow (I will handle buildAndPushRelease.py and smokeTestRelease.py if they
> need fixes but I'm not sure about the releaseWizard.*).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9809) Adapt Release Wizard to only release Lucene
[ https://issues.apache.org/jira/browse/LUCENE-9809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17433424#comment-17433424 ] Jan Høydahl commented on LUCENE-9809: - I'll try the env.var approach for pw. If folks are OK with pulling the maven artifacts from git repo's build folders instead of using exactlyt the same folder that was uploaded and voted upon, then this should be ok. I guess the theoretical risk is if RM gets confused and somehow invokes another build and uploads another commit hash to maven. As long as the releaseWizard is used, which has a dedicated git clone this should not be a real risk anyway. > Adapt Release Wizard to only release Lucene > --- > > Key: LUCENE-9809 > URL: https://issues.apache.org/jira/browse/LUCENE-9809 > Project: Lucene - Core > Issue Type: Sub-task >Reporter: Jan Høydahl >Assignee: Jan Høydahl >Priority: Major > Fix For: main (9.0) > > Time Spent: 1h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Comment Edited] (LUCENE-9809) Adapt Release Wizard to only release Lucene
[ https://issues.apache.org/jira/browse/LUCENE-9809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17433424#comment-17433424 ] Jan Høydahl edited comment on LUCENE-9809 at 10/24/21, 11:16 AM: - I'll try the env.var approach for pw. If folks are OK with pulling the maven artifacts from git repo's build folders instead of using exactlyt the same folder that was uploaded and voted upon, then this should be ok. I guess the theoretical risk is if RM gets confused and somehow invokes another build and uploads another commit hash to maven. There will be several days in between the first build, voting and the staging to artifactory. But as long as the releaseWizard is used, which has a dedicated git clone, this should not be a real risk anyway. was (Author: janhoy): I'll try the env.var approach for pw. If folks are OK with pulling the maven artifacts from git repo's build folders instead of using exactlyt the same folder that was uploaded and voted upon, then this should be ok. I guess the theoretical risk is if RM gets confused and somehow invokes another build and uploads another commit hash to maven. As long as the releaseWizard is used, which has a dedicated git clone this should not be a real risk anyway. > Adapt Release Wizard to only release Lucene > --- > > Key: LUCENE-9809 > URL: https://issues.apache.org/jira/browse/LUCENE-9809 > Project: Lucene - Core > Issue Type: Sub-task >Reporter: Jan Høydahl >Assignee: Jan Høydahl >Priority: Major > Fix For: main (9.0) > > Time Spent: 1h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-10192) Drop third-party JARs from the binary distribution
[ https://issues.apache.org/jira/browse/LUCENE-10192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17433425#comment-17433425 ] Dawid Weiss commented on LUCENE-10192: -- ([~janhoy] - I didn't touch the smoke tester yet; will do it before comitting if we agree the new format is ok). > Drop third-party JARs from the binary distribution > -- > > Key: LUCENE-10192 > URL: https://issues.apache.org/jira/browse/LUCENE-10192 > Project: Lucene - Core > Issue Type: Task >Reporter: Dawid Weiss >Assignee: Dawid Weiss >Priority: Trivial > Time Spent: 10m > Remaining Estimate: 0h > > [~janhoy] Are we ready (with respect to scripts) for this change? I'd like to > do it but I'm not sure whether the release wizard doesn't depend on it > somehow (I will handle buildAndPushRelease.py and smokeTestRelease.py if they > need fixes but I'm not sure about the releaseWizard.*). -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9809) Adapt Release Wizard to only release Lucene
[ https://issues.apache.org/jira/browse/LUCENE-9809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17433426#comment-17433426 ] Dawid Weiss commented on LUCENE-9809: - The alternative is to create a "bundle" which is a ZIP of all the artifacts for Nexus and upload it manually. Then you wouldn't need to provide any passwords at all - you'd create that bundle and the RM would have to upload and accept them in Nexus. Don't know what is more convenient. > Adapt Release Wizard to only release Lucene > --- > > Key: LUCENE-9809 > URL: https://issues.apache.org/jira/browse/LUCENE-9809 > Project: Lucene - Core > Issue Type: Sub-task >Reporter: Jan Høydahl >Assignee: Jan Høydahl >Priority: Major > Fix For: main (9.0) > > Time Spent: 1h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9660) gradle task cache should not cache --tests
[
https://issues.apache.org/jira/browse/LUCENE-9660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17433428#comment-17433428
]
Robert Muir commented on LUCENE-9660:
-
So we should be using {{cleanTest}} always?
There's never a time when i ask gradle to run tests, and don't want it to
actually in fact, run the test.
> gradle task cache should not cache --tests
> --
>
> Key: LUCENE-9660
> URL: https://issues.apache.org/jira/browse/LUCENE-9660
> Project: Lucene - Core
> Issue Type: Improvement
> Components: general/build
>Reporter: David Smiley
>Assignee: Dawid Weiss
>Priority: Minor
>
> I recently ran a specific test at the CLI via gradle to see if a particular
> build failure repeats. It includes the {{--tests}} command line option to
> specify the test. The test passed. Later I wanted to run it again; I
> suspected it might be flakey. Gradle completed in 10 seconds, and I'm
> certain it didn't actually run the test. There was no printout and the
> build/test-results/test/outputs/... from the test run still had not changed
> from previously.
> Mike Drob informed me of "gradlew cleanTest" but I'd prefer to not have to
> know about that, at least not for the specific case of wanting to execute a
> specific test.
> CC [~dweiss]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9660) gradle task cache should not cache --tests
[
https://issues.apache.org/jira/browse/LUCENE-9660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17433436#comment-17433436
]
Dawid Weiss commented on LUCENE-9660:
-
No, no. Normally tests will execute every time - the random seed is changing
from run to run, classes change, etc. If you rerun the same Gradle test task
(say, fix the seed and don't touch anything else) it will be up to date and
will be skipped. This scenario is so uncommon that I don't think it requires
special treatment. If you do encounter it, cleanTest first to force the test
task to run again
> gradle task cache should not cache --tests
> --
>
> Key: LUCENE-9660
> URL: https://issues.apache.org/jira/browse/LUCENE-9660
> Project: Lucene - Core
> Issue Type: Improvement
> Components: general/build
>Reporter: David Smiley
>Assignee: Dawid Weiss
>Priority: Minor
>
> I recently ran a specific test at the CLI via gradle to see if a particular
> build failure repeats. It includes the {{--tests}} command line option to
> specify the test. The test passed. Later I wanted to run it again; I
> suspected it might be flakey. Gradle completed in 10 seconds, and I'm
> certain it didn't actually run the test. There was no printout and the
> build/test-results/test/outputs/... from the test run still had not changed
> from previously.
> Mike Drob informed me of "gradlew cleanTest" but I'd prefer to not have to
> know about that, at least not for the specific case of wanting to execute a
> specific test.
> CC [~dweiss]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9660) gradle task cache should not cache --tests
[
https://issues.apache.org/jira/browse/LUCENE-9660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17433443#comment-17433443
]
Robert Muir commented on LUCENE-9660:
-
[~dweiss] It is pretty common when reproducing test failures from jenkins.
Often times people want to use threads from tests, also various things about
lucene (e.g. ConcurrentMergeScheduler) aren't reproducible by default.
So it is pretty common to have to run a seed multiple times to make it
reproduce.
> gradle task cache should not cache --tests
> --
>
> Key: LUCENE-9660
> URL: https://issues.apache.org/jira/browse/LUCENE-9660
> Project: Lucene - Core
> Issue Type: Improvement
> Components: general/build
>Reporter: David Smiley
>Assignee: Dawid Weiss
>Priority: Minor
>
> I recently ran a specific test at the CLI via gradle to see if a particular
> build failure repeats. It includes the {{--tests}} command line option to
> specify the test. The test passed. Later I wanted to run it again; I
> suspected it might be flakey. Gradle completed in 10 seconds, and I'm
> certain it didn't actually run the test. There was no printout and the
> build/test-results/test/outputs/... from the test run still had not changed
> from previously.
> Mike Drob informed me of "gradlew cleanTest" but I'd prefer to not have to
> know about that, at least not for the specific case of wanting to execute a
> specific test.
> CC [~dweiss]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9660) gradle task cache should not cache --tests
[
https://issues.apache.org/jira/browse/LUCENE-9660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17433444#comment-17433444
]
Robert Muir commented on LUCENE-9660:
-
I still think my argument holds, what is the use-case for *not* running the
test, ever?
It seems we're defending gradle's broken behavior here instead of configuring
it to our needs?
> gradle task cache should not cache --tests
> --
>
> Key: LUCENE-9660
> URL: https://issues.apache.org/jira/browse/LUCENE-9660
> Project: Lucene - Core
> Issue Type: Improvement
> Components: general/build
>Reporter: David Smiley
>Assignee: Dawid Weiss
>Priority: Minor
>
> I recently ran a specific test at the CLI via gradle to see if a particular
> build failure repeats. It includes the {{--tests}} command line option to
> specify the test. The test passed. Later I wanted to run it again; I
> suspected it might be flakey. Gradle completed in 10 seconds, and I'm
> certain it didn't actually run the test. There was no printout and the
> build/test-results/test/outputs/... from the test run still had not changed
> from previously.
> Mike Drob informed me of "gradlew cleanTest" but I'd prefer to not have to
> know about that, at least not for the specific case of wanting to execute a
> specific test.
> CC [~dweiss]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9660) gradle task cache should not cache --tests
[
https://issues.apache.org/jira/browse/LUCENE-9660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17433445#comment-17433445
]
Dawid Weiss commented on LUCENE-9660:
-
It's not broken - it's normal and consistent with any other task (that is
incremental). If your inputs and outputs are up-to-date, the task does not run.
Lucene tests use random seed as input so they're sort of special here but other
projects would use this feature to skip tests that have passed if nothing has
changed in between.
If you're using gradle for a longer while there is nothing surprising about
this. Changing this behavior would violate the principle of least surprise, in
my opinion... But if you insist you can change it by forcing task outputs to
never be up to date -
{code}
outputs.upToDateWhen { false }
{code}
https://docs.gradle.org/current/javadoc/org/gradle/api/tasks/TaskOutputs.html#upToDateWhen-groovy.lang.Closure-
This should be added to each and every test task - somewhere here, perhaps:
https://github.com/dweiss/lucene/blob/main/gradle/testing/defaults-tests.gradle#L88
> gradle task cache should not cache --tests
> --
>
> Key: LUCENE-9660
> URL: https://issues.apache.org/jira/browse/LUCENE-9660
> Project: Lucene - Core
> Issue Type: Improvement
> Components: general/build
>Reporter: David Smiley
>Assignee: Dawid Weiss
>Priority: Minor
>
> I recently ran a specific test at the CLI via gradle to see if a particular
> build failure repeats. It includes the {{--tests}} command line option to
> specify the test. The test passed. Later I wanted to run it again; I
> suspected it might be flakey. Gradle completed in 10 seconds, and I'm
> certain it didn't actually run the test. There was no printout and the
> build/test-results/test/outputs/... from the test run still had not changed
> from previously.
> Mike Drob informed me of "gradlew cleanTest" but I'd prefer to not have to
> know about that, at least not for the specific case of wanting to execute a
> specific test.
> CC [~dweiss]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9809) Adapt Release Wizard to only release Lucene
[ https://issues.apache.org/jira/browse/LUCENE-9809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17433446#comment-17433446 ] Dawid Weiss commented on LUCENE-9809: - https://help.sonatype.com/repomanager2/staging-releases/artifact-bundles#ArtifactBundles-UploadinganArtifactBundle This is technically easy to do. Actually, it could be a viable and simpler alternative - it doesn't require passwords and makes the upload process more explicit. I've used it in the past to speed up Sonatype uploads too (because oss nexus was pretty slow). Up to you, Jan. I can tweak the code to prepare such a bundle instead of the artifact folder. > Adapt Release Wizard to only release Lucene > --- > > Key: LUCENE-9809 > URL: https://issues.apache.org/jira/browse/LUCENE-9809 > Project: Lucene - Core > Issue Type: Sub-task >Reporter: Jan Høydahl >Assignee: Jan Høydahl >Priority: Major > Fix For: main (9.0) > > Time Spent: 1h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Reopened] (LUCENE-9660) gradle task cache should not cache --tests
[
https://issues.apache.org/jira/browse/LUCENE-9660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dawid Weiss reopened LUCENE-9660:
-
Reopening to provide a workaround since there is a discussion about it.
> gradle task cache should not cache --tests
> --
>
> Key: LUCENE-9660
> URL: https://issues.apache.org/jira/browse/LUCENE-9660
> Project: Lucene - Core
> Issue Type: Improvement
> Components: general/build
>Reporter: David Smiley
>Assignee: Dawid Weiss
>Priority: Minor
>
> I recently ran a specific test at the CLI via gradle to see if a particular
> build failure repeats. It includes the {{--tests}} command line option to
> specify the test. The test passed. Later I wanted to run it again; I
> suspected it might be flakey. Gradle completed in 10 seconds, and I'm
> certain it didn't actually run the test. There was no printout and the
> build/test-results/test/outputs/... from the test run still had not changed
> from previously.
> Mike Drob informed me of "gradlew cleanTest" but I'd prefer to not have to
> know about that, at least not for the specific case of wanting to execute a
> specific test.
> CC [~dweiss]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-10192) Drop third-party JARs from the binary distribution
[
https://issues.apache.org/jira/browse/LUCENE-10192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17433457#comment-17433457
]
Dawid Weiss commented on LUCENE-10192:
--
Correction - the classpath element is not needed. We just need to add log4j api
to the root module set (because lucene.luke doesn't declare it).
{code:java}
java --module-path modules;modules-thirdparty --add-modules
org.apache.logging.log4j --module lucene.luke
{code}
No need for any version-specific information, yay.
> Drop third-party JARs from the binary distribution
> --
>
> Key: LUCENE-10192
> URL: https://issues.apache.org/jira/browse/LUCENE-10192
> Project: Lucene - Core
> Issue Type: Task
>Reporter: Dawid Weiss
>Assignee: Dawid Weiss
>Priority: Trivial
> Time Spent: 10m
> Remaining Estimate: 0h
>
> [~janhoy] Are we ready (with respect to scripts) for this change? I'd like to
> do it but I'm not sure whether the release wizard doesn't depend on it
> somehow (I will handle buildAndPushRelease.py and smokeTestRelease.py if they
> need fixes but I'm not sure about the releaseWizard.*).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-10197) UnifiedHighlighter should use builders for thread-safety
[ https://issues.apache.org/jira/browse/LUCENE-10197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17433506#comment-17433506 ] Animesh Pandey commented on LUCENE-10197: - Thanks for feedback on the patch. I am fine with the PR process. I actually thought it was only used by committers. I will create a PR in the next iteration even though I don't think it would include all the changes. Since development would be going on after Nov 2021, I think it should be fine it goes in 2023 but I'll try to make progress as quickly as possible. > UnifiedHighlighter should use builders for thread-safety > > > Key: LUCENE-10197 > URL: https://issues.apache.org/jira/browse/LUCENE-10197 > Project: Lucene - Core > Issue Type: Improvement > Components: modules/highlighter >Reporter: Animesh Pandey >Priority: Minor > Labels: newdev > Attachments: LUCENE-10197.patch > > > UnifiedHighlighter is not thread-safe due to the presence of setters. We can > move the fields to builder so that the class becomes thread-safe. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene] dsmiley commented on pull request #353: LUCENE-9488 Assemble source tar, with checksum and signing
dsmiley commented on pull request #353:
URL: https://github.com/apache/lucene/pull/353#issuecomment-950407268
I'm looking at this and seeing `gradle/maven/publications-maven.gradle` at
the bottom has:
```
tasks.withType(GenerateModuleMetadata) {
enabled = false
}
```
Added in this PR by @dweiss shown by git blame.
Why? This means if we use more sophisticated metadata like
[feature-variants](https://docs.gradle.org/current/userguide/feature_variants.html)
(which I'm working on now for spatial-extras), it will not be published, which
is a shame.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Created] (LUCENE-10202) Expose dependencies using Gradle Feature Variants
David Smiley created LUCENE-10202: - Summary: Expose dependencies using Gradle Feature Variants Key: LUCENE-10202 URL: https://issues.apache.org/jira/browse/LUCENE-10202 Project: Lucene - Core Issue Type: Improvement Components: modules/spatial-extras Reporter: David Smiley Assignee: David Smiley The spatial-extras module has several dependencies. However some of them like spatial3d (aka Geo3d) are only needed for certain features. Likewise, JTS could be exposed here as well, and should be opt-in. In Maven, these should be "optional". Gradle has a cool alternative for Gradle consumers to select named "feature variants" this module could expose so that it doesn't have to pick the right dependency versions. https://docs.gradle.org/current/userguide/feature_variants.html -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-10202) Expose dependencies using Gradle Feature Variants
[ https://issues.apache.org/jira/browse/LUCENE-10202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17433521#comment-17433521 ] David Smiley commented on LUCENE-10202: --- [~daddywri] when I was working on the PR to name the "feature variant" implemented by the spatial3d module, I was contemplating what to name it. I should probably pick the obvious name – "spatial3d" after the dependency it brings in. But its name had me thinking (and we've been over this in the past) about how difficult it is to name that module. "3d" gives people the wrong impression. When I tell people about that module, I say it has "surface of a sphere geometry" (and yes it does ellepsoid, I know). Thus perhaps "spatial-spherical" might have been a better name? > Expose dependencies using Gradle Feature Variants > - > > Key: LUCENE-10202 > URL: https://issues.apache.org/jira/browse/LUCENE-10202 > Project: Lucene - Core > Issue Type: Improvement > Components: modules/spatial-extras >Reporter: David Smiley >Assignee: David Smiley >Priority: Minor > > The spatial-extras module has several dependencies. However some of them > like spatial3d (aka Geo3d) are only needed for certain features. Likewise, > JTS could be exposed here as well, and should be opt-in. In Maven, these > should be "optional". Gradle has a cool alternative for Gradle consumers to > select named "feature variants" this module could expose so that it doesn't > have to pick the right dependency versions. > https://docs.gradle.org/current/userguide/feature_variants.html -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (LUCENE-10202) spatial: expose dependencies using Gradle Feature Variants
[ https://issues.apache.org/jira/browse/LUCENE-10202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Smiley updated LUCENE-10202: -- Summary: spatial: expose dependencies using Gradle Feature Variants (was: Expose dependencies using Gradle Feature Variants) > spatial: expose dependencies using Gradle Feature Variants > -- > > Key: LUCENE-10202 > URL: https://issues.apache.org/jira/browse/LUCENE-10202 > Project: Lucene - Core > Issue Type: Improvement > Components: modules/spatial-extras >Reporter: David Smiley >Assignee: David Smiley >Priority: Minor > > The spatial-extras module has several dependencies. However some of them > like spatial3d (aka Geo3d) are only needed for certain features. Likewise, > JTS could be exposed here as well, and should be opt-in. In Maven, these > should be "optional". Gradle has a cool alternative for Gradle consumers to > select named "feature variants" this module could expose so that it doesn't > have to pick the right dependency versions. > https://docs.gradle.org/current/userguide/feature_variants.html -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene] dsmiley opened a new pull request #411: LUCENE-10202: spatial: feature-variants
dsmiley opened a new pull request #411: URL: https://github.com/apache/lucene/pull/411 https://issues.apache.org/jira/browse/LUCENE-10202 TODO on choice of naming RE spherical vs spatial3d Note: as of this writing, the build doesn't publish the nice Gradle metadata -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene] dsmiley commented on pull request #411: LUCENE-10202: spatial: feature-variants
dsmiley commented on pull request #411: URL: https://github.com/apache/lucene/pull/411#issuecomment-950427225 I spot-checked the generated Maven POMs locally. As expected and according to the Gradle docs, these dependencies showed as optional. I see something weird though -- this dependency: ``` org.apache lucene-root 9.0.0-SNAPSHOT compile true ``` This weird lucene-root dependency is only here; it isn't in any of the other Lucene modules. I confirmed this only appears here because of the use of this Gradle feature. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene] apanimesh061 opened a new pull request #412: LUCENE-10197: UnifiedHighlighter should use builders for thread-safety
apanimesh061 opened a new pull request #412: URL: https://github.com/apache/lucene/pull/412 # Description `UnifiedHighlighter` is not thread-safe due to the presence of setters. We can move the fields to builder so that the class becomes thread-safe. Objective is to made `UnifiedHighlighter` immutable. # Solution Add a builder that is able to set all fields of the `UnifiedHighlighter` class. # Tests ``` ./gradlew -p lucene/highlighter test ./gradlew check ``` # Checklist Please review the following and check all that apply: - [X] I have reviewed the guidelines for [How to Contribute](https://wiki.apache.org/lucene/HowToContribute) and my code conforms to the standards described there to the best of my ability. - [X] I have created a Jira issue and added the issue ID to my pull request title. - [X] I have given Lucene maintainers [access](https://help.github.com/en/articles/allowing-changes-to-a-pull-request-branch-created-from-a-fork) to contribute to my PR branch. (optional but recommended) - [X ] I have developed this patch against the `main` branch. - [X] I have run `./gradlew check`. - [X] I have added tests for my changes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-10197) UnifiedHighlighter should use builders for thread-safety
[ https://issues.apache.org/jira/browse/LUCENE-10197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17433549#comment-17433549 ] Animesh Pandey commented on LUCENE-10197: - [~dsmiley] I've created a draft PR for you to look at: https://github.com/apache/lucene/pull/412. Please take a look whenever you get a chance. I've addressed some of your previous comments. > UnifiedHighlighter should use builders for thread-safety > > > Key: LUCENE-10197 > URL: https://issues.apache.org/jira/browse/LUCENE-10197 > Project: Lucene - Core > Issue Type: Improvement > Components: modules/highlighter >Reporter: Animesh Pandey >Priority: Minor > Labels: newdev > Attachments: LUCENE-10197.patch > > Time Spent: 10m > Remaining Estimate: 0h > > UnifiedHighlighter is not thread-safe due to the presence of setters. We can > move the fields to builder so that the class becomes thread-safe. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-10191) Optimize vector functions by precomputing magnitudes
[
https://issues.apache.org/jira/browse/LUCENE-10191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17433565#comment-17433565
]
Julie Tibshirani commented on LUCENE-10191:
---
This is helpful feedback. I'm also sensitive to the fact that the more
complexity we add to a format, the harder it is for BWC and for future
implementations.
Some background: I think supporting Euclidean distance is really important.
With certain datasets, similarity is measured in terms of Euclidean distance
(instead of cosine), and in these cases it's critical to use Euclidean to get
sensible results. Cosine similarity is less critical, since we could ask users
to normalize all vectors to unit length before indexing + searching, and use
dot product. Personally I think cosine is valuable (more details in
https://issues.apache.org/jira/browse/LUCENE-10146), but am very happy to
discuss trade-offs. In general, supporting different vector functions is low
complexity compared to the ANN data structure itself.
{quote}Instead, slower functions needing different representation should really
be different codecs... And trying to support these functions the way it happens
now is wrong to do and will lead to hairballs.
{quote}
To check I understand the idea – are you suggesting a separate format per ANN
method, per similarity function?
> Optimize vector functions by precomputing magnitudes
>
>
> Key: LUCENE-10191
> URL: https://issues.apache.org/jira/browse/LUCENE-10191
> Project: Lucene - Core
> Issue Type: Improvement
>Reporter: Julie Tibshirani
>Priority: Minor
>
> Both euclidean distance (L2 norm) and cosine similarity can be expressed in
> terms of dot product and vector magnitudes:
> * l2_norm(a, b) = ||a - b|| = sqrt(||a||^2 - 2(a . b) + ||b||^2)
> * cosine(a, b) = a . b / ||a|| ||b||
> We could compute and store each vector's magnitude upfront while indexing,
> and compute the query vector's magnitude once per query. Then we'd calculate
> the distance using our (very optimized) dot product method, plus the
> precomputed values.
> This is an exploratory issue: I haven't tested this out yet, so I'm not sure
> how much it would help. I would at least expect it to help with cosine
> similarity – several months ago we tried out similar ideas in Elasticsearch
> and were able to get a nice boost in cosine performance.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Comment Edited] (LUCENE-10191) Optimize vector functions by precomputing magnitudes
[
https://issues.apache.org/jira/browse/LUCENE-10191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17433565#comment-17433565
]
Julie Tibshirani edited comment on LUCENE-10191 at 10/25/21, 5:29 AM:
--
This is helpful feedback. I'm also sensitive to the fact that the more
complexity we add to a format, the harder it is for BWC and for future
implementations.
Some background: I think supporting Euclidean distance is really important.
With certain datasets, similarity is measured in terms of Euclidean distance
(instead of cosine), and in these cases it's critical to use Euclidean to get
sensible results. There's not a way to transform the data so you can just use
dot product. Cosine similarity is less critical, since we could ask users to
normalize all vectors to unit length before indexing + searching, and use dot
product. Personally I think cosine is valuable (more details in
https://issues.apache.org/jira/browse/LUCENE-10146), but am very happy to
discuss trade-offs. In general, supporting different vector functions is low
complexity compared to the ANN data structure itself.
{quote}Instead, slower functions needing different representation should really
be different codecs... And trying to support these functions the way it happens
now is wrong to do and will lead to hairballs.
{quote}
To check I understand the idea – are you suggesting a separate format per ANN
method, per similarity function?
was (Author: julietibs):
This is helpful feedback. I'm also sensitive to the fact that the more
complexity we add to a format, the harder it is for BWC and for future
implementations.
Some background: I think supporting Euclidean distance is really important.
With certain datasets, similarity is measured in terms of Euclidean distance
(instead of cosine), and in these cases it's critical to use Euclidean to get
sensible results. Cosine similarity is less critical, since we could ask users
to normalize all vectors to unit length before indexing + searching, and use
dot product. Personally I think cosine is valuable (more details in
https://issues.apache.org/jira/browse/LUCENE-10146), but am very happy to
discuss trade-offs. In general, supporting different vector functions is low
complexity compared to the ANN data structure itself.
{quote}Instead, slower functions needing different representation should really
be different codecs... And trying to support these functions the way it happens
now is wrong to do and will lead to hairballs.
{quote}
To check I understand the idea – are you suggesting a separate format per ANN
method, per similarity function?
> Optimize vector functions by precomputing magnitudes
>
>
> Key: LUCENE-10191
> URL: https://issues.apache.org/jira/browse/LUCENE-10191
> Project: Lucene - Core
> Issue Type: Improvement
>Reporter: Julie Tibshirani
>Priority: Minor
>
> Both euclidean distance (L2 norm) and cosine similarity can be expressed in
> terms of dot product and vector magnitudes:
> * l2_norm(a, b) = ||a - b|| = sqrt(||a||^2 - 2(a . b) + ||b||^2)
> * cosine(a, b) = a . b / ||a|| ||b||
> We could compute and store each vector's magnitude upfront while indexing,
> and compute the query vector's magnitude once per query. Then we'd calculate
> the distance using our (very optimized) dot product method, plus the
> precomputed values.
> This is an exploratory issue: I haven't tested this out yet, so I'm not sure
> how much it would help. I would at least expect it to help with cosine
> similarity – several months ago we tried out similar ideas in Elasticsearch
> and were able to get a nice boost in cosine performance.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene] dweiss commented on pull request #353: LUCENE-9488 Assemble source tar, with checksum and signing
dweiss commented on pull request #353: URL: https://github.com/apache/lucene/pull/353#issuecomment-950585341 Because back in the day it didn't work for me and broke downstream maven builds. If you'd like to check whether things work without it, please remove it and file a pr. I don't care. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene] dweiss commented on pull request #411: LUCENE-10202: spatial: feature-variants
dweiss commented on pull request #411: URL: https://github.com/apache/lucene/pull/411#issuecomment-950586805 > This weird lucene-root dependency is only here; it isn't in any of the other Lucene modules. I confirmed this only appears here because of the use of this Gradle feature. Things like this is exactly what prevented me from using metadata. If you have a dependency on such a weird-looking artifact and it's not published anywhere, things break (even if it's optional). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene] dweiss commented on pull request #411: LUCENE-10202: spatial: feature-variants
dweiss commented on pull request #411: URL: https://github.com/apache/lucene/pull/411#issuecomment-950587825 So, in other words - the POMs have to be right. If enabling metadata means we have odd entries in POMs, I'm -1 for adding metadata. The lucene-root dependency may be a result of how palantir works internally too - I'm really not sure and I can't devote any time to look into this (I did in the past, without success). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
