[GitHub] [lucene] gf2121 commented on a diff in pull request #12574: Make TaskExecutor public

2023-09-20 Thread via GitHub
gf2121 commented on code in PR #12574: URL: https://github.com/apache/lucene/pull/12574#discussion_r1331468750 ## lucene/core/src/java/org/apache/lucene/search/TaskExecutor.java: ## @@ -85,11 +85,24 @@ final List invokeAll(Collection> tasks) throws IOException { return re

[GitHub] [lucene] gf2121 opened a new pull request, #12573: Speed up sort on deleted terms

2023-09-20 Thread via GitHub
gf2121 opened a new pull request, #12573: URL: https://github.com/apache/lucene/pull/12573 ### Description Recently, we captured a flame graph in a scene with frequent updates, which showed that sorting deleted terms occupied a high CPU ratio. Currently, we use JDK sort to sort

[GitHub] [lucene] shubhamvishu commented on pull request #12183: Make TermStates#build concurrent

2023-09-20 Thread via GitHub
shubhamvishu commented on PR #12183: URL: https://github.com/apache/lucene/pull/12183#issuecomment-1727638896 > Hey @shubhamvishu heads up: I merged #12659 to address the deadlock issue and opened #12574 to adjust TaskExecutor visibility outside of this PR. Hopefully you are next going to b

[GitHub] [lucene] javanna commented on pull request #12183: Make TermStates#build concurrent

2023-09-20 Thread via GitHub
javanna commented on PR #12183: URL: https://github.com/apache/lucene/pull/12183#issuecomment-1727419929 Hey @shubhamvishu heads up: I merged #12659 to address the deadlock issue and opened #12574 to adjust TaskExecutor visibility outside of this PR. Hopefully you are next going to be able

[GitHub] [lucene] benwtrent closed issue #12570: Reading after Segment Merge fails for HNSW

2023-09-20 Thread via GitHub
benwtrent closed issue #12570: Reading after Segment Merge fails for HNSW URL: https://github.com/apache/lucene/issues/12570 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

[GitHub] [lucene] javanna merged pull request #12574: Make TaskExecutor public

2023-09-20 Thread via GitHub
javanna merged PR #12574: URL: https://github.com/apache/lucene/pull/12574 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucene.apa

[GitHub] [lucene] jpountz opened a new issue, #12572: Make IndexWriter#flushNextBuffer flush deletes too?

2023-09-20 Thread via GitHub
jpountz opened a new issue, #12572: URL: https://github.com/apache/lucene/issues/12572 ### Description `IndexWriter#flushNextBuffer()` is a convenient way to control indexing buffer sizes across multiple index writers. Unfortunately, it seems that it only ever flushes DWPTs, and neve

[GitHub] [lucene] javanna opened a new pull request, #12574: Make TaskExecutor public

2023-09-20 Thread via GitHub
javanna opened a new pull request, #12574: URL: https://github.com/apache/lucene/pull/12574 TaskExecutor is currently package private. We have scenarios where we want to parallelize the execution and reuse it outside of its package, hence this commit makes it public. Note that its co

[GitHub] [lucene] zhaih closed issue #9660: ArrayIndexOutOfBoundsException in ByteBlockPool [LUCENE-8614]

2023-09-20 Thread via GitHub
zhaih closed issue #9660: ArrayIndexOutOfBoundsException in ByteBlockPool [LUCENE-8614] URL: https://github.com/apache/lucene/issues/9660 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [lucene] stefanvodita commented on issue #9660: ArrayIndexOutOfBoundsException in ByteBlockPool [LUCENE-8614]

2023-09-20 Thread via GitHub
stefanvodita commented on issue #9660: URL: https://github.com/apache/lucene/issues/9660#issuecomment-1727196480 Yes, it's resolved. Thanks, Patrick! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[GitHub] [lucene] javanna commented on pull request #12569: Prevent concurrent tasks from parallelizing further

2023-09-20 Thread via GitHub
javanna commented on PR #12569: URL: https://github.com/apache/lucene/pull/12569#issuecomment-1727388144 > It might be worth using CallerRunsPolicy with a small queue in tests sometimes, as this is an interesting case that will make tasks run in the current thread. Given that TaskEx

[GitHub] [lucene] kaivalnp opened a new issue, #12575: Allow implementers of AbstractKnnVectorQuery to access final topK results?

2023-09-20 Thread via GitHub
kaivalnp opened a new issue, #12575: URL: https://github.com/apache/lucene/issues/12575 ### Context Vector search is performed in [`AbstractKnnVectorQuery`](https://github.com/kaivalnp/lucene/blob/main/lucene/core/src/java/org/apache/lucene/search/AbstractKnnVectorQuery.java), where

[GitHub] [lucene] javanna commented on a diff in pull request #12574: Make TaskExecutor public

2023-09-20 Thread via GitHub
javanna commented on code in PR #12574: URL: https://github.com/apache/lucene/pull/12574#discussion_r1331497358 ## lucene/core/src/java/org/apache/lucene/search/TaskExecutor.java: ## @@ -85,11 +85,24 @@ final List invokeAll(Collection> tasks) throws IOException { return r

[GitHub] [lucene] javanna commented on a diff in pull request #12569: Prevent concurrent tasks from parallelizing further

2023-09-20 Thread via GitHub
javanna commented on code in PR #12569: URL: https://github.com/apache/lucene/pull/12569#discussion_r1331259326 ## lucene/core/src/java/org/apache/lucene/search/TaskExecutor.java: ## @@ -22,18 +22,29 @@ import java.util.Collection; import java.util.List; import java.util.Obje

[GitHub] [lucene] javanna commented on a diff in pull request #12569: Prevent concurrent tasks from parallelizing further

2023-09-20 Thread via GitHub
javanna commented on code in PR #12569: URL: https://github.com/apache/lucene/pull/12569#discussion_r1331266192 ## lucene/core/src/java/org/apache/lucene/search/TaskExecutor.java: ## @@ -22,18 +22,28 @@ import java.util.Collection; import java.util.List; import java.util.Obje

[GitHub] [lucene] zhaih commented on issue #9660: ArrayIndexOutOfBoundsException in ByteBlockPool [LUCENE-8614]

2023-09-20 Thread via GitHub
zhaih commented on issue #9660: URL: https://github.com/apache/lucene/issues/9660#issuecomment-1727052393 @stefanvodita Seems the issue is resolved? I closed the issue, feel free to reopen it -- This is an automated message from the Apache Git Service. To respond to the message, please lo

[GitHub] [lucene] javanna merged pull request #12569: Prevent concurrent tasks from parallelizing further

2023-09-20 Thread via GitHub
javanna merged PR #12569: URL: https://github.com/apache/lucene/pull/12569 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucene.apa

[GitHub] [lucene] vsop-479 commented on pull request #12528: Early terminate visit BKD leaf when current value greater than upper point in sorted dim.

2023-09-20 Thread via GitHub
vsop-479 commented on PR #12528: URL: https://github.com/apache/lucene/pull/12528#issuecomment-1727215669 @jpountz Please take a look when you get a chance! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

[GitHub] [lucene] shubhamvishu commented on pull request #12183: Make TermStates#build concurrent

2023-09-20 Thread via GitHub
shubhamvishu commented on PR #12183: URL: https://github.com/apache/lucene/pull/12183#issuecomment-1727782959 I have rebased the PR based on the changes in #12574. Could some please take a look? Thanks! -- This is an automated message from the Apache Git Service. To respond to the message

[GitHub] [lucene] javanna commented on pull request #12569: Prevent concurrent tasks from parallelizing further

2023-09-20 Thread via GitHub
javanna commented on PR #12569: URL: https://github.com/apache/lucene/pull/12569#issuecomment-172724 I pushed new commits to address the latest review comments, thanks for all the input. This should be ready now. -- This is an automated message from the Apache Git Service. To respond

[GitHub] [lucene] tylerbertrand opened a new pull request, #12577: Resolve CompileJava task cache miss

2023-09-20 Thread via GitHub
tylerbertrand opened a new pull request, #12577: URL: https://github.com/apache/lucene/pull/12577 ### Description Resolves `CompileJava` cache miss caused by `options.compilerArgs` input difference. Moved the `apijar` input file to a `CommandLineArgumentProvider` to apply rela

[GitHub] [lucene] javanna opened a new pull request, #12578: Deprecate IndexSearcher#getExecutor

2023-09-20 Thread via GitHub
javanna opened a new pull request, #12578: URL: https://github.com/apache/lucene/pull/12578 We have recently introduced a TaskExecutor abstraction, which is meant to be used to execute concurrent tasks using the executor provided to the IndexSearcher constructor. All concurrenct tasks shoul

[GitHub] [lucene] uschindler commented on a diff in pull request #12569: Prevent concurrent tasks from parallelizing further

2023-09-20 Thread via GitHub
uschindler commented on code in PR #12569: URL: https://github.com/apache/lucene/pull/12569#discussion_r1331117098 ## lucene/core/src/java/org/apache/lucene/search/TaskExecutor.java: ## @@ -22,18 +22,29 @@ import java.util.Collection; import java.util.List; import java.util.O

[GitHub] [lucene] javanna commented on a diff in pull request #12569: Prevent concurrent tasks from parallelizing further

2023-09-20 Thread via GitHub
javanna commented on code in PR #12569: URL: https://github.com/apache/lucene/pull/12569#discussion_r1331266192 ## lucene/core/src/java/org/apache/lucene/search/TaskExecutor.java: ## @@ -22,18 +22,28 @@ import java.util.Collection; import java.util.List; import java.util.Obje

[GitHub] [lucene] kaivalnp opened a new issue, #12579: [DISCUSS] Should there be a threshold-based vector search API?

2023-09-20 Thread via GitHub
kaivalnp opened a new issue, #12579: URL: https://github.com/apache/lucene/issues/12579 ### Context Almost all [vector search algorithms](https://ann-benchmarks.com/index.html#algorithms) focus on getting the `topK` results for a given query vector. This however, may not be the best

[GitHub] [lucene] dweiss commented on a diff in pull request #12577: Resolve CompileJava task cache miss

2023-09-20 Thread via GitHub
dweiss commented on code in PR #12577: URL: https://github.com/apache/lucene/pull/12577#discussion_r1331954883 ## gradle/java/core-mrjar.gradle: ## @@ -29,20 +29,19 @@ configure(project(":lucene:core")) { dependencies.add("main${jdkVersion}Implementation", sourceSets.mai

[GitHub] [lucene] javanna commented on pull request #12578: Deprecate IndexSearcher#getExecutor

2023-09-20 Thread via GitHub
javanna commented on PR #12578: URL: https://github.com/apache/lucene/pull/12578#issuecomment-1728197364 > I assume you will remove the deprecated method in main branch and add an entry to the MIGRATE.txt there? Yes. but I'll open a PR against main to have people double check the chan

[GitHub] [lucene] tylerbertrand commented on a diff in pull request #12577: Resolve CompileJava task cache miss

2023-09-20 Thread via GitHub
tylerbertrand commented on code in PR #12577: URL: https://github.com/apache/lucene/pull/12577#discussion_r1332210752 ## gradle/java/core-mrjar.gradle: ## @@ -29,20 +29,19 @@ configure(project(":lucene:core")) { dependencies.add("main${jdkVersion}Implementation", sourceS

[GitHub] [lucene] tylerbertrand commented on a diff in pull request #12577: Resolve CompileJava task cache miss

2023-09-20 Thread via GitHub
tylerbertrand commented on code in PR #12577: URL: https://github.com/apache/lucene/pull/12577#discussion_r1332210752 ## gradle/java/core-mrjar.gradle: ## @@ -29,20 +29,19 @@ configure(project(":lucene:core")) { dependencies.add("main${jdkVersion}Implementation", sourceS