[GitHub] [lucene] alessandrobenedetti commented on issue #11507: Increase the number of dims for KNN vectors to 2048 [LUCENE-10471]

2023-06-29 Thread via GitHub
alessandrobenedetti commented on issue #11507: URL: https://github.com/apache/lucene/issues/11507#issuecomment-1612704311 I am extremely curious, what should we consider a good performance to index <3M docs? I mean, I agree we should always try to improve things and aim for the stars, bu

[GitHub] [lucene] mikemccand commented on issue #12399: Would SIMD powered sort (on top of Panama) be worth it?

2023-06-29 Thread via GitHub
mikemccand commented on issue #12399: URL: https://github.com/apache/lucene/issues/12399#issuecomment-1612734709 OK thanks for all the exciting discussion. I agree we should focus on only the white-hot spots given how risking using Panama vector APIs is (trappy performance pitfalls). Let'

[GitHub] [lucene] mikemccand closed issue #12399: Would SIMD powered sort (on top of Panama) be worth it?

2023-06-29 Thread via GitHub
mikemccand closed issue #12399: Would SIMD powered sort (on top of Panama) be worth it? URL: https://github.com/apache/lucene/issues/12399 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specifi

[GitHub] [lucene] mikemccand commented on issue #12399: Would SIMD powered sort (on top of Panama) be worth it?

2023-06-29 Thread via GitHub
mikemccand commented on issue #12399: URL: https://github.com/apache/lucene/issues/12399#issuecomment-161294 > > Yeah, some of our custom sorts are because we want to sort one array, but use the sort key from another parallel array. Unfortunately I don't think (?) the JDK has existing A

[GitHub] [lucene] easyice commented on a diff in pull request #12381: Speed up NumericDocValuesWriter with index sorting

2023-06-29 Thread via GitHub
easyice commented on code in PR #12381: URL: https://github.com/apache/lucene/pull/12381#discussion_r1246447022 ## lucene/core/src/java/org/apache/lucene/index/DocsWithFieldSet.java: ## @@ -75,4 +75,9 @@ public DocIdSetIterator iterator() { public int cardinality() { ret

[GitHub] [lucene] alessandrobenedetti commented on issue #12313: Multi-value Support for KnnVectorField

2023-06-29 Thread via GitHub
alessandrobenedetti commented on issue #12313: URL: https://github.com/apache/lucene/issues/12313#issuecomment-1612857327 I'll have to spend more brain time on the proposed block-join alternative, but isn't it already "available" in that form? (with the consequent problems and benefits of j

[GitHub] [lucene] easyice commented on a diff in pull request #12381: Speed up NumericDocValuesWriter with index sorting

2023-06-29 Thread via GitHub
easyice commented on code in PR #12381: URL: https://github.com/apache/lucene/pull/12381#discussion_r1246457143 ## lucene/core/src/java/org/apache/lucene/util/BitSet.java: ## @@ -111,4 +111,70 @@ public void or(DocIdSetIterator iter) throws IOException { set(doc); }

[GitHub] [lucene] easyice commented on pull request #12381: Speed up NumericDocValuesWriter with index sorting

2023-06-29 Thread via GitHub
easyice commented on PR #12381: URL: https://github.com/apache/lucene/pull/12381#issuecomment-1612872730 @jpountz Thank you for comments, it's very helpful to me, the code has updated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [lucene] benwtrent commented on issue #12313: Multi-value Support for KnnVectorField

2023-06-29 Thread via GitHub
benwtrent commented on issue #12313: URL: https://github.com/apache/lucene/issues/12313#issuecomment-1612949170 > I'll have to spend more brain time on the proposed block-join alternative, but isn't it already "available" in that form? (with the consequent problems and benefits of joins?)

[GitHub] [lucene] jpountz commented on a diff in pull request #12381: Speed up NumericDocValuesWriter with index sorting

2023-06-29 Thread via GitHub
jpountz commented on code in PR #12381: URL: https://github.com/apache/lucene/pull/12381#discussion_r1246499123 ## lucene/core/src/java/org/apache/lucene/index/NumericDocValuesWriter.java: ## @@ -114,7 +121,12 @@ static DocValuesProducer getDocValuesProducer( final NumericD

[GitHub] [lucene] uschindler opened a new pull request, #12404: Remove and add some legacy java.util classes to forbiddenapis (Stack, Hashtable, Vector)

2023-06-29 Thread via GitHub
uschindler opened a new pull request, #12404: URL: https://github.com/apache/lucene/pull/12404 I noticed on some other PR that some classes (especially `java.util.Stack`) are used in Lucene code although they are really legacy from Java 1.0. The problem with them is that they are synchroniz

[GitHub] [lucene] uschindler commented on a diff in pull request #12314: Multi-value support for KnnVectorField

2023-06-29 Thread via GitHub
uschindler commented on code in PR #12314: URL: https://github.com/apache/lucene/pull/12314#discussion_r1246537955 ## lucene/core/src/java/org/apache/lucene/index/DocsWithVectorsSet.java: ## @@ -0,0 +1,112 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or

[GitHub] [lucene] jpountz commented on a diff in pull request #12383: Assign a dummy simScorer in TermsWeight if score is not needed

2023-06-29 Thread via GitHub
jpountz commented on code in PR #12383: URL: https://github.com/apache/lucene/pull/12383#discussion_r1246588872 ## lucene/core/src/test/org/apache/lucene/search/TestTermQuery.java: ## @@ -164,6 +170,57 @@ public void testGetTermStates() throws Exception { IOUtils.close(read

[GitHub] [lucene] cbismuth commented on issue #11754: TestBoolean2.testRandomQueries fails in CI due to eating up heap space

2023-06-29 Thread via GitHub
cbismuth commented on issue #11754: URL: https://github.com/apache/lucene/issues/11754#issuecomment-1613312521 Can't reproduce against `main` branch, maybe not relevant anymore? ``` MAC-0320:lucene cbismuth$ ./gradlew -p lucene/core -Ptests.seed=A0883CC08C1C22AB -Ptests.heapsize=25

[GitHub] [lucene] LuXugang opened a new pull request, #12405: Skip docs with Docvalues in NumericLeafComparator

2023-06-29 Thread via GitHub
LuXugang opened a new pull request, #12405: URL: https://github.com/apache/lucene/pull/12405 ### Description Could we implement `TermOrdValLeafComparator`'s same logic that using NumericDocValues to skip docs in `NumericLeafComparator` if we could not get a iterator by bkd? -- Thi

[GitHub] [lucene] LuXugang commented on pull request #12405: Skip docs with Docvalues in NumericLeafComparator

2023-06-29 Thread via GitHub
LuXugang commented on PR #12405: URL: https://github.com/apache/lucene/pull/12405#issuecomment-1613330925 ```java public void test() throws IOException { final Directory dir = newDirectory(); IndexWriterConfig config = new IndexWriterConfig()

[GitHub] [lucene] uschindler merged pull request #12404: Remove usage and add some legacy java.util classes to forbiddenapis (Stack, Hashtable, Vector)

2023-06-29 Thread via GitHub
uschindler merged PR #12404: URL: https://github.com/apache/lucene/pull/12404 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucene.

[GitHub] [lucene] almogtavor closed issue #12318: Async Usage of Lucene Monitor through a Reactive Programming based application

2023-06-29 Thread via GitHub
almogtavor closed issue #12318: Async Usage of Lucene Monitor through a Reactive Programming based application URL: https://github.com/apache/lucene/issues/12318 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL abo

[GitHub] [lucene] almogtavor commented on issue #12318: Async Usage of Lucene Monitor through a Reactive Programming based application

2023-06-29 Thread via GitHub
almogtavor commented on issue #12318: URL: https://github.com/apache/lucene/issues/12318#issuecomment-1613462613 I see. Thanks alot for the help -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to th

[GitHub] [lucene] almogtavor opened a new issue, #12406: Register nested queries (ToParentBlockJoinQuery) to Lucene Monitor

2023-06-29 Thread via GitHub
almogtavor opened a new issue, #12406: URL: https://github.com/apache/lucene/issues/12406 ### Description I use Lucene Monitor with regular `Document` objects and it works just fine. The thing is that I'd like to match with Solr queries that I use in a nested collection, but I've cou

[GitHub] [lucene] sgup432 commented on a diff in pull request #12383: Assign a dummy simScorer in TermsWeight if score is not needed

2023-06-29 Thread via GitHub
sgup432 commented on code in PR #12383: URL: https://github.com/apache/lucene/pull/12383#discussion_r1246890555 ## lucene/core/src/test/org/apache/lucene/search/TestTermQuery.java: ## @@ -164,6 +170,57 @@ public void testGetTermStates() throws Exception { IOUtils.close(read

[GitHub] [lucene] sohami commented on pull request #12374: Add CachingLeafSlicesSupplier to compute the LeafSlices for concurrent segment search

2023-06-29 Thread via GitHub
sohami commented on PR #12374: URL: https://github.com/apache/lucene/pull/12374#issuecomment-1613524919 > LGTM. Can you add a CHANGES entry under 9.8? Added. I have also rebased the changes. Thanks @jpountz for your time and guidance. -- This is an automated message from the Apache

[GitHub] [lucene] sgup432 commented on pull request #12383: Assign a dummy simScorer in TermsWeight if score is not needed

2023-06-29 Thread via GitHub
sgup432 commented on PR #12383: URL: https://github.com/apache/lucene/pull/12383#issuecomment-1613526593 > I left minor suggestions but it looks good to me. Can you add a CHANGES entry under 9.8? @jpountz Sure. I have added a CHANGE entry. -- This is an automated message from the A

[GitHub] [lucene] easyice commented on a diff in pull request #12381: Speed up NumericDocValuesWriter with index sorting

2023-06-29 Thread via GitHub
easyice commented on code in PR #12381: URL: https://github.com/apache/lucene/pull/12381#discussion_r1247414106 ## lucene/core/src/java/org/apache/lucene/index/NormValuesWriter.java: ## @@ -76,7 +76,8 @@ public void flush(SegmentWriteState state, Sorter.DocMap sortMap, NormsCon

[GitHub] [lucene] easyice commented on a diff in pull request #12381: Speed up NumericDocValuesWriter with index sorting

2023-06-29 Thread via GitHub
easyice commented on code in PR #12381: URL: https://github.com/apache/lucene/pull/12381#discussion_r1247416965 ## lucene/core/src/java/org/apache/lucene/index/NumericDocValuesWriter.java: ## @@ -234,10 +246,41 @@ public long cost() { static class NumericDVs { private fi