[GitHub] [lucene] tylerbertrand commented on a diff in pull request #12577: Resolve CompileJava task cache miss

2023-09-20 Thread via GitHub
tylerbertrand commented on code in PR #12577: URL: https://github.com/apache/lucene/pull/12577#discussion_r1332210752 ## gradle/java/core-mrjar.gradle: ## @@ -29,20 +29,19 @@ configure(project(":lucene:core")) { dependencies.add("main${jdkVersion}Implementation", sourceS

[GitHub] [lucene] tylerbertrand commented on a diff in pull request #12577: Resolve CompileJava task cache miss

2023-09-20 Thread via GitHub
tylerbertrand commented on code in PR #12577: URL: https://github.com/apache/lucene/pull/12577#discussion_r1332210752 ## gradle/java/core-mrjar.gradle: ## @@ -29,20 +29,19 @@ configure(project(":lucene:core")) { dependencies.add("main${jdkVersion}Implementation", sourceS

[GitHub] [lucene] javanna commented on pull request #12578: Deprecate IndexSearcher#getExecutor

2023-09-20 Thread via GitHub
javanna commented on PR #12578: URL: https://github.com/apache/lucene/pull/12578#issuecomment-1728197364 > I assume you will remove the deprecated method in main branch and add an entry to the MIGRATE.txt there? Yes. but I'll open a PR against main to have people double check the chan

[GitHub] [lucene] dweiss commented on a diff in pull request #12577: Resolve CompileJava task cache miss

2023-09-20 Thread via GitHub
dweiss commented on code in PR #12577: URL: https://github.com/apache/lucene/pull/12577#discussion_r1331954883 ## gradle/java/core-mrjar.gradle: ## @@ -29,20 +29,19 @@ configure(project(":lucene:core")) { dependencies.add("main${jdkVersion}Implementation", sourceSets.mai

[GitHub] [lucene] kaivalnp opened a new issue, #12579: [DISCUSS] Should there be a threshold-based vector search API?

2023-09-20 Thread via GitHub
kaivalnp opened a new issue, #12579: URL: https://github.com/apache/lucene/issues/12579 ### Context Almost all [vector search algorithms](https://ann-benchmarks.com/index.html#algorithms) focus on getting the `topK` results for a given query vector. This however, may not be the best

[GitHub] [lucene] javanna commented on a diff in pull request #12569: Prevent concurrent tasks from parallelizing further

2023-09-20 Thread via GitHub
javanna commented on code in PR #12569: URL: https://github.com/apache/lucene/pull/12569#discussion_r1331266192 ## lucene/core/src/java/org/apache/lucene/search/TaskExecutor.java: ## @@ -22,18 +22,28 @@ import java.util.Collection; import java.util.List; import java.util.Obje

[GitHub] [lucene] uschindler commented on a diff in pull request #12569: Prevent concurrent tasks from parallelizing further

2023-09-20 Thread via GitHub
uschindler commented on code in PR #12569: URL: https://github.com/apache/lucene/pull/12569#discussion_r1331117098 ## lucene/core/src/java/org/apache/lucene/search/TaskExecutor.java: ## @@ -22,18 +22,29 @@ import java.util.Collection; import java.util.List; import java.util.O

[GitHub] [lucene] javanna opened a new pull request, #12578: Deprecate IndexSearcher#getExecutor

2023-09-20 Thread via GitHub
javanna opened a new pull request, #12578: URL: https://github.com/apache/lucene/pull/12578 We have recently introduced a TaskExecutor abstraction, which is meant to be used to execute concurrent tasks using the executor provided to the IndexSearcher constructor. All concurrenct tasks shoul

[GitHub] [lucene] tylerbertrand opened a new pull request, #12577: Resolve CompileJava task cache miss

2023-09-20 Thread via GitHub
tylerbertrand opened a new pull request, #12577: URL: https://github.com/apache/lucene/pull/12577 ### Description Resolves `CompileJava` cache miss caused by `options.compilerArgs` input difference. Moved the `apijar` input file to a `CommandLineArgumentProvider` to apply rela

[GitHub] [lucene] javanna commented on pull request #12569: Prevent concurrent tasks from parallelizing further

2023-09-20 Thread via GitHub
javanna commented on PR #12569: URL: https://github.com/apache/lucene/pull/12569#issuecomment-172724 I pushed new commits to address the latest review comments, thanks for all the input. This should be ready now. -- This is an automated message from the Apache Git Service. To respond

[GitHub] [lucene] shubhamvishu commented on pull request #12183: Make TermStates#build concurrent

2023-09-20 Thread via GitHub
shubhamvishu commented on PR #12183: URL: https://github.com/apache/lucene/pull/12183#issuecomment-1727782959 I have rebased the PR based on the changes in #12574. Could some please take a look? Thanks! -- This is an automated message from the Apache Git Service. To respond to the message

[GitHub] [lucene] vsop-479 commented on pull request #12528: Early terminate visit BKD leaf when current value greater than upper point in sorted dim.

2023-09-20 Thread via GitHub
vsop-479 commented on PR #12528: URL: https://github.com/apache/lucene/pull/12528#issuecomment-1727215669 @jpountz Please take a look when you get a chance! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

[GitHub] [lucene] javanna merged pull request #12569: Prevent concurrent tasks from parallelizing further

2023-09-20 Thread via GitHub
javanna merged PR #12569: URL: https://github.com/apache/lucene/pull/12569 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucene.apa

[GitHub] [lucene] zhaih commented on issue #9660: ArrayIndexOutOfBoundsException in ByteBlockPool [LUCENE-8614]

2023-09-20 Thread via GitHub
zhaih commented on issue #9660: URL: https://github.com/apache/lucene/issues/9660#issuecomment-1727052393 @stefanvodita Seems the issue is resolved? I closed the issue, feel free to reopen it -- This is an automated message from the Apache Git Service. To respond to the message, please lo

[GitHub] [lucene] javanna commented on a diff in pull request #12569: Prevent concurrent tasks from parallelizing further

2023-09-20 Thread via GitHub
javanna commented on code in PR #12569: URL: https://github.com/apache/lucene/pull/12569#discussion_r1331266192 ## lucene/core/src/java/org/apache/lucene/search/TaskExecutor.java: ## @@ -22,18 +22,28 @@ import java.util.Collection; import java.util.List; import java.util.Obje

[GitHub] [lucene] javanna commented on a diff in pull request #12569: Prevent concurrent tasks from parallelizing further

2023-09-20 Thread via GitHub
javanna commented on code in PR #12569: URL: https://github.com/apache/lucene/pull/12569#discussion_r1331259326 ## lucene/core/src/java/org/apache/lucene/search/TaskExecutor.java: ## @@ -22,18 +22,29 @@ import java.util.Collection; import java.util.List; import java.util.Obje

[GitHub] [lucene] javanna commented on a diff in pull request #12574: Make TaskExecutor public

2023-09-20 Thread via GitHub
javanna commented on code in PR #12574: URL: https://github.com/apache/lucene/pull/12574#discussion_r1331497358 ## lucene/core/src/java/org/apache/lucene/search/TaskExecutor.java: ## @@ -85,11 +85,24 @@ final List invokeAll(Collection> tasks) throws IOException { return r

[GitHub] [lucene] kaivalnp opened a new issue, #12575: Allow implementers of AbstractKnnVectorQuery to access final topK results?

2023-09-20 Thread via GitHub
kaivalnp opened a new issue, #12575: URL: https://github.com/apache/lucene/issues/12575 ### Context Vector search is performed in [`AbstractKnnVectorQuery`](https://github.com/kaivalnp/lucene/blob/main/lucene/core/src/java/org/apache/lucene/search/AbstractKnnVectorQuery.java), where

[GitHub] [lucene] javanna commented on pull request #12569: Prevent concurrent tasks from parallelizing further

2023-09-20 Thread via GitHub
javanna commented on PR #12569: URL: https://github.com/apache/lucene/pull/12569#issuecomment-1727388144 > It might be worth using CallerRunsPolicy with a small queue in tests sometimes, as this is an interesting case that will make tasks run in the current thread. Given that TaskEx

[GitHub] [lucene] stefanvodita commented on issue #9660: ArrayIndexOutOfBoundsException in ByteBlockPool [LUCENE-8614]

2023-09-20 Thread via GitHub
stefanvodita commented on issue #9660: URL: https://github.com/apache/lucene/issues/9660#issuecomment-1727196480 Yes, it's resolved. Thanks, Patrick! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[GitHub] [lucene] zhaih closed issue #9660: ArrayIndexOutOfBoundsException in ByteBlockPool [LUCENE-8614]

2023-09-20 Thread via GitHub
zhaih closed issue #9660: ArrayIndexOutOfBoundsException in ByteBlockPool [LUCENE-8614] URL: https://github.com/apache/lucene/issues/9660 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [lucene] javanna opened a new pull request, #12574: Make TaskExecutor public

2023-09-20 Thread via GitHub
javanna opened a new pull request, #12574: URL: https://github.com/apache/lucene/pull/12574 TaskExecutor is currently package private. We have scenarios where we want to parallelize the execution and reuse it outside of its package, hence this commit makes it public. Note that its co

[GitHub] [lucene] jpountz opened a new issue, #12572: Make IndexWriter#flushNextBuffer flush deletes too?

2023-09-20 Thread via GitHub
jpountz opened a new issue, #12572: URL: https://github.com/apache/lucene/issues/12572 ### Description `IndexWriter#flushNextBuffer()` is a convenient way to control indexing buffer sizes across multiple index writers. Unfortunately, it seems that it only ever flushes DWPTs, and neve

[GitHub] [lucene] javanna merged pull request #12574: Make TaskExecutor public

2023-09-20 Thread via GitHub
javanna merged PR #12574: URL: https://github.com/apache/lucene/pull/12574 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucene.apa

[GitHub] [lucene] benwtrent closed issue #12570: Reading after Segment Merge fails for HNSW

2023-09-20 Thread via GitHub
benwtrent closed issue #12570: Reading after Segment Merge fails for HNSW URL: https://github.com/apache/lucene/issues/12570 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

[GitHub] [lucene] javanna commented on pull request #12183: Make TermStates#build concurrent

2023-09-20 Thread via GitHub
javanna commented on PR #12183: URL: https://github.com/apache/lucene/pull/12183#issuecomment-1727419929 Hey @shubhamvishu heads up: I merged #12659 to address the deadlock issue and opened #12574 to adjust TaskExecutor visibility outside of this PR. Hopefully you are next going to be able

[GitHub] [lucene] shubhamvishu commented on pull request #12183: Make TermStates#build concurrent

2023-09-20 Thread via GitHub
shubhamvishu commented on PR #12183: URL: https://github.com/apache/lucene/pull/12183#issuecomment-1727638896 > Hey @shubhamvishu heads up: I merged #12659 to address the deadlock issue and opened #12574 to adjust TaskExecutor visibility outside of this PR. Hopefully you are next going to b

[GitHub] [lucene] gf2121 opened a new pull request, #12573: Speed up sort on deleted terms

2023-09-20 Thread via GitHub
gf2121 opened a new pull request, #12573: URL: https://github.com/apache/lucene/pull/12573 ### Description Recently, we captured a flame graph in a scene with frequent updates, which showed that sorting deleted terms occupied a high CPU ratio. Currently, we use JDK sort to sort

[GitHub] [lucene] gf2121 commented on a diff in pull request #12574: Make TaskExecutor public

2023-09-20 Thread via GitHub
gf2121 commented on code in PR #12574: URL: https://github.com/apache/lucene/pull/12574#discussion_r1331468750 ## lucene/core/src/java/org/apache/lucene/search/TaskExecutor.java: ## @@ -85,11 +85,24 @@ final List invokeAll(Collection> tasks) throws IOException { return re