Re: [PR] Avoid unnecessary comparison for CELL_CROSSES_QUERY cases [lucene]

2025-05-22 Thread via GitHub
iverase commented on code in PR #14626: URL: https://github.com/apache/lucene/pull/14626#discussion_r2103918228 ## lucene/core/src/java/org/apache/lucene/search/PointRangeQuery.java: ## @@ -154,16 +154,18 @@ private Relation relate(byte[] minPackedValue, byte[] maxPackedValue)

Re: [PR] Avoid unnecessary comparison for CELL_CROSSES_QUERY cases [lucene]

2025-05-22 Thread via GitHub
iverase commented on code in PR #14626: URL: https://github.com/apache/lucene/pull/14626#discussion_r2103918228 ## lucene/core/src/java/org/apache/lucene/search/PointRangeQuery.java: ## @@ -154,16 +154,18 @@ private Relation relate(byte[] minPackedValue, byte[] maxPackedValue)

Re: [PR] Add Query for reranking KnnFloatVectorQuery with full-precision vectors [lucene]

2025-05-22 Thread via GitHub
dungba88 commented on PR #14009: URL: https://github.com/apache/lucene/pull/14009#issuecomment-2903403699 Let me rephrase it better. I'm not worrying about the computation cost, as you mentioned they are the same. In Amazon (where both Vigya and I are working at), we usually run KnnF

Re: [PR] Add Query for reranking KnnFloatVectorQuery with full-precision vectors [lucene]

2025-05-22 Thread via GitHub
dungba88 commented on code in PR #14009: URL: https://github.com/apache/lucene/pull/14009#discussion_r2103857961 ## lucene/core/src/java/org/apache/lucene/search/RerankFloatVectorQuery.java: ## @@ -0,0 +1,116 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one

Re: [PR] Add Query for reranking KnnFloatVectorQuery with full-precision vectors [lucene]

2025-05-22 Thread via GitHub
vigyasharma commented on PR #14009: URL: https://github.com/apache/lucene/pull/14009#issuecomment-2903244426 > For the alternative implementation we leave that up to the final hit collection to return the top-k based on a ranking mechanism Those final TopDocs collectors will use the s

Re: [PR] Add Query for reranking KnnFloatVectorQuery with full-precision vectors [lucene]

2025-05-22 Thread via GitHub
vigyasharma commented on code in PR #14009: URL: https://github.com/apache/lucene/pull/14009#discussion_r2103784830 ## lucene/core/src/java/org/apache/lucene/search/RerankFloatVectorQuery.java: ## @@ -0,0 +1,116 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under o

Re: [I] Segment count (merging) can impact recall on KNN ParentJoin queries [lucene]

2025-05-22 Thread via GitHub
dungba88 commented on issue #14643: URL: https://github.com/apache/lucene/issues/14643#issuecomment-2903095335 Not sure if this is relevant: https://github.com/mikemccand/luceneutil/issues/385, but if it's the same problem it will be fixed with latest luceneutil. -- This is an automated

Re: [PR] Add Query for reranking KnnFloatVectorQuery with full-precision vectors [lucene]

2025-05-22 Thread via GitHub
dungba88 commented on PR #14009: URL: https://github.com/apache/lucene/pull/14009#issuecomment-2903029304 > This gets handled by the k you pass to searcher.search(), and the top docs collector you use. It's same as any other query. I guess the main difference is when the results got t

Re: [PR] Avoid unnecessary comparison for CELL_CROSSES_QUERY cases [lucene]

2025-05-22 Thread via GitHub
github-actions[bot] commented on PR #14626: URL: https://github.com/apache/lucene/pull/14626#issuecomment-2902936990 This PR has not had activity in the past 2 weeks, labeling it as stale. If the PR is waiting for review, notify the d...@lucene.apache.org list. Thank you for your contributi

Re: [PR] Add Query for reranking KnnFloatVectorQuery with full-precision vectors [lucene]

2025-05-22 Thread via GitHub
msokolov commented on PR #14009: URL: https://github.com/apache/lucene/pull/14009#issuecomment-2902557094 Is it this? https://github.com/apache/lucene/blob/main/lucene/queries/src/java/org/apache/lucene/queries/function/FunctionScoreQuery.java -- This is an automated message from the

Re: [PR] Add Query for reranking KnnFloatVectorQuery with full-precision vectors [lucene]

2025-05-22 Thread via GitHub
vigyasharma commented on PR #14009: URL: https://github.com/apache/lucene/pull/14009#issuecomment-2902502742 > Lucene typically rewrites the Query first, and then createWeight on the simplest form. That's a good callout. I'll override the `rewrite` method to rewrite the inner query.

Re: [PR] Add Query for reranking KnnFloatVectorQuery with full-precision vectors [lucene]

2025-05-22 Thread via GitHub
msokolov commented on PR #14009: URL: https://github.com/apache/lucene/pull/14009#issuecomment-2902339944 I haven't dug in deep yet, but along the lines of further generality, would it make sense to accept a DoubleValuesSource (that can compute dot products against a vector field) rather t

Re: [PR] [WIP] Multi-Vector support for HNSW search [lucene]

2025-05-22 Thread via GitHub
heemin32 commented on PR #13525: URL: https://github.com/apache/lucene/pull/13525#issuecomment-2902300025 On a second thought, if user just diable indexing, implementing multi vector in the same field might be same as having a separate field without indexing capability. -- This is an aut

Re: [I] Segment count (merging) can impact recall on KNN ParentJoin queries [lucene]

2025-05-22 Thread via GitHub
msokolov commented on issue #14643: URL: https://github.com/apache/lucene/issues/14643#issuecomment-2902293560 > Hmm it's odd for the 500K docs case that recall is so much better with FEWER segments: .969 with 8 segments, .705 with 89 segments. -- This is an automated message from the Apa

Re: [I] Integrate a JVector codec for KNN searches [lucene]

2025-05-22 Thread via GitHub
msokolov commented on issue #14681: URL: https://github.com/apache/lucene/issues/14681#issuecomment-2902282294 This seems worth exploring. One question I have, after reading the linked discussion thread about integrating JVector as an OpenSearch plugin, is about this claim ("JVector is a th

Re: [PR] Use a temporary repository location to download certain ecj versions ("drops") [lucene]

2025-05-22 Thread via GitHub
dweiss merged PR #14703: URL: https://github.com/apache/lucene/pull/14703 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucene.apac

Re: [PR] [WIP] Multi-Vector support for HNSW search [lucene]

2025-05-22 Thread via GitHub
heemin32 commented on PR #13525: URL: https://github.com/apache/lucene/pull/13525#issuecomment-2902248141 I believe this proposal to add a new field for multi-vector support is facing significant challenges primarily because we aim to support HNSW-based search on it. However, if our goal we

Re: [PR] [WIP] Multi-Vector support for HNSW search [lucene]

2025-05-22 Thread via GitHub
heemin32 commented on PR #13525: URL: https://github.com/apache/lucene/pull/13525#issuecomment-2902247646 Certainly! Here's a clearer and more polished rephrasing of your message: --- I believe this proposal to add a new field for multi-vector support is facing significant chal

Re: [PR] Allow reading binary doc values as a RandomAccessInput [lucene]

2025-05-22 Thread via GitHub
github-actions[bot] commented on PR #13948: URL: https://github.com/apache/lucene/pull/13948#issuecomment-2902069576 This PR does not have an entry in lucene/CHANGES.txt. Consider adding one. If the PR doesn't need a changelog entry, then add the skip-changelog-check label to it and you wil

Re: [I] Remove Telugu normalization of vu వు to ma మ from IndicNormalizer [lucene]

2025-05-22 Thread via GitHub
Trey314159 commented on issue #14659: URL: https://github.com/apache/lucene/issues/14659#issuecomment-2902054358 @praveen-d291: Thanks for the pull request! I was unsure how best to modify the tests since I don't read Telugu. I couldn't tell what would make natural-looking examples and I di

Re: [PR] Update the IOContext on IndexInput rather than the ReadAdvice [lucene]

2025-05-22 Thread via GitHub
jpountz commented on code in PR #14702: URL: https://github.com/apache/lucene/pull/14702#discussion_r2102989752 ## lucene/core/src/java/org/apache/lucene/codecs/KnnVectorsReader.java: ## @@ -124,7 +124,7 @@ public abstract void search( * * The default implementation retu

Re: [PR] Add assumption to ignore test failures due to disconnected graphs [lucene]

2025-05-22 Thread via GitHub
msokolov merged PR #14696: URL: https://github.com/apache/lucene/pull/14696 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucene.ap

Re: [PR] Return MatchNoDocsQuery when IndexOrDocValuesQuery::rewrite does not match [lucene]

2025-05-22 Thread via GitHub
ChrisHegarty merged PR #14700: URL: https://github.com/apache/lucene/pull/14700 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucen

Re: [PR] Fix AbstractRangeQueryNode#toQueryString [lucene]

2025-05-22 Thread via GitHub
phb-ig commented on code in PR #14697: URL: https://github.com/apache/lucene/pull/14697#discussion_r2102606331 ## lucene/queryparser/src/test/org/apache/lucene/queryparser/flexible/standard/nodes/TestAbstractRangeQueryNode.java: ## @@ -0,0 +1,69 @@ +/* + * Licensed to the Apache

Re: [PR] Fix AbstractRangeQueryNode#toQueryString [lucene]

2025-05-22 Thread via GitHub
phb-ig commented on code in PR #14697: URL: https://github.com/apache/lucene/pull/14697#discussion_r2102594352 ## lucene/CHANGES.txt: ## @@ -854,7 +862,7 @@ Improvements * GITHUB#13285: Early terminate graph searches of AbstractVectorSimilarityQuery to follow timeout set fro

Re: [PR] Fix AbstractRangeQueryNode#toQueryString [lucene]

2025-05-22 Thread via GitHub
phb-ig commented on PR #14697: URL: https://github.com/apache/lucene/pull/14697#issuecomment-2901267587 @stefanvodita I am the one who originally submitted the patches all that time ago because the work I was doing at the time required inspecting the query string and altering, then re-parsi

Re: [PR] Add Query for reranking KnnFloatVectorQuery with full-precision vectors [lucene]

2025-05-22 Thread via GitHub
dungba88 commented on PR #14009: URL: https://github.com/apache/lucene/pull/14009#issuecomment-2901240220 I changed the query to be generic but keep other the same. (Didn't have JDK24 yet so build would fail.) -- This is an automated message from the Apache Git Service. To respond to the

Re: [PR] Add Query for reranking KnnFloatVectorQuery with full-precision vectors [lucene]

2025-05-22 Thread via GitHub
github-actions[bot] commented on PR #14009: URL: https://github.com/apache/lucene/pull/14009#issuecomment-2901186441 This PR does not have an entry in lucene/CHANGES.txt. Consider adding one. If the PR doesn't need a changelog entry, then add the skip-changelog-check label to it and you wil

Re: [PR] Use a temporary repository location to download certain ecj versions ("drops") [lucene]

2025-05-22 Thread via GitHub
github-actions[bot] commented on PR #14703: URL: https://github.com/apache/lucene/pull/14703#issuecomment-2901110046 This PR does not have an entry in lucene/CHANGES.txt. Consider adding one. If the PR doesn't need a changelog entry, then add the skip-changelog-check label to it and you wil

Re: [PR] Add Query for reranking KnnFloatVectorQuery with full-precision vectors [lucene]

2025-05-22 Thread via GitHub
dungba88 commented on PR #14009: URL: https://github.com/apache/lucene/pull/14009#issuecomment-2901114417 No worry, I think generalization makes sense to me too. And I like the idea of moving this forward. The use of `KnnFloatVectorQuery` in this implementation only adds a small bene

[PR] Use a temporary repository location to download certain ecj versions ("drops") [lucene]

2025-05-22 Thread via GitHub
dweiss opened a new pull request, #14703: URL: https://github.com/apache/lucene/pull/14703 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-ma

Re: [PR] Update the IOContext on IndexInput rather than the ReadAdvice [lucene]

2025-05-22 Thread via GitHub
github-actions[bot] commented on PR #14702: URL: https://github.com/apache/lucene/pull/14702#issuecomment-2900966501 This PR does not have an entry in lucene/CHANGES.txt. Consider adding one. If the PR doesn't need a changelog entry, then add the skip-changelog-check label to it and you wil

[PR] Update the IOContext on IndexInput rather than the ReadAdvice [lucene]

2025-05-22 Thread via GitHub
thecoop opened a new pull request, #14702: URL: https://github.com/apache/lucene/pull/14702 Update the `IOContext` on `IndexInput` implementations rather than the `ReadAdvice`. This change means that `ReadAdvice` is now only used by `MMapDirectory` and friends. Note that this is a refactori

Re: [PR] Speed up conjunctive queries that need scores. [lucene]

2025-05-22 Thread via GitHub
jpountz commented on PR #14690: URL: https://github.com/apache/lucene/pull/14690#issuecomment-2900934920 I'm superseding this change with a more general one for now, which doesn't introduce new public APIs: #14701. We can look into taking ideas from this PR as follow-ups. -- This is an a

Re: [PR] Speed up conjunctive queries that need scores. [lucene]

2025-05-22 Thread via GitHub
jpountz closed pull request #14690: Speed up conjunctive queries that need scores. URL: https://github.com/apache/lucene/pull/14690 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comme

Re: [PR] Refactor main top-n bulk scorers to evaluate hits in a more term-at-a-time fashion. [lucene]

2025-05-22 Thread via GitHub
jpountz commented on PR #14701: URL: https://github.com/apache/lucene/pull/14701#issuecomment-2900931305 Filtered queries get a slowdown, but some important queries get a big speedup: ``` TaskQPS baseline StdDevQPS my_modified_version StdDev

Re: [PR] Refactor main top-n bulk scorers to evaluate hits in a more term-at-a-time fashion. [lucene]

2025-05-22 Thread via GitHub
github-actions[bot] commented on PR #14701: URL: https://github.com/apache/lucene/pull/14701#issuecomment-2900925674 This PR does not have an entry in lucene/CHANGES.txt. Consider adding one. If the PR doesn't need a changelog entry, then add the skip-changelog-check label to it and you wil

[PR] Refactor main top-n bulk scorers to evaluate hits in a more term-at-a-time fashion. [lucene]

2025-05-22 Thread via GitHub
jpountz opened a new pull request, #14701: URL: https://github.com/apache/lucene/pull/14701 `MaxScoreBulkScorer` and `BlockMaxConjunctionBulkScorer` currently evaluate hits in a doc-at-a-time (DAAT) fashion, meaning that they they look at all their clauses to find the next doc and so forth

Re: [I] Remove Telugu normalization of vu వు to ma మ from IndicNormalizer [lucene]

2025-05-22 Thread via GitHub
rmuir commented on issue #14659: URL: https://github.com/apache/lucene/issues/14659#issuecomment-2900766946 The filter here is working as documented: problem is, user didn't read the documentation. Just don't use the filter if you dont want the transformations that it does. -- This is an

[PR] Fixed incorrect Telugu normalization of vu వు to ma మ ( [lucene]

2025-05-22 Thread via GitHub
praveen-d291 opened a new pull request, #14699: URL: https://github.com/apache/lucene/pull/14699 Fixes: #14659 Remove incorrect Telugu వు/మ conflation in Indic Normalization. They look similar, but they are distinct with different meanings. Currently "వు" is mapped to "మ" in

Re: [I] Remove Telugu normalization of vu వు to ma మ from IndicNormalizer [lucene]

2025-05-22 Thread via GitHub
rmuir commented on issue #14659: URL: https://github.com/apache/lucene/issues/14659#issuecomment-2900761266 > It's like conflating "rn" and "m" to merge burn/bum and corn/com. It could happen when reading quickly or with poor handwriting, but it is not something that should happen for searc

Re: [I] Nightly benchmark regression on 2025.05.01 [lucene]

2025-05-22 Thread via GitHub
mikemccand commented on issue #14630: URL: https://github.com/apache/lucene/issues/14630#issuecomment-2900807348 Hmm @yugushihuang (on my team (Amazon product search) team) found this is another way to query the kernel (our Amazon Linux 2 boxes seem not to have `/proc/config.gz`): ``

[PR] Return MatchNoDocsQuery when IndexOrDocValuesQuery::rewrite does not match [lucene]

2025-05-22 Thread via GitHub
ChrisHegarty opened a new pull request, #14700: URL: https://github.com/apache/lucene/pull/14700 In a similar way to how `IndexOrDocValuesQuery` may rewrite to a `MatchAllDocsQuery`, if either of the queries rewrites to match no docs, then return a `MatchNoDocsQuery`. -- This is an autom

Re: [PR] Return MatchNoDocsQuery when IndexOrDocValuesQuery::rewrite does not match [lucene]

2025-05-22 Thread via GitHub
github-actions[bot] commented on PR #14700: URL: https://github.com/apache/lucene/pull/14700#issuecomment-2900736002 This PR does not have an entry in lucene/CHANGES.txt. Consider adding one. If the PR doesn't need a changelog entry, then add the skip-changelog-check label to it and you wil

Re: [PR] Minor access modifier adjustment to a couple of lucene90 backward compat types [lucene]

2025-05-22 Thread via GitHub
ChrisHegarty merged PR #14695: URL: https://github.com/apache/lucene/pull/14695 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucen

Re: [PR] Speed up exhaustive evaluation. [lucene]

2025-05-22 Thread via GitHub
jpountz merged PR #14679: URL: https://github.com/apache/lucene/pull/14679 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucene.apa

Re: [PR] Fixed incorrect Telugu normalization of vu వు to ma మ ( [lucene]

2025-05-22 Thread via GitHub
github-actions[bot] commented on PR #14699: URL: https://github.com/apache/lucene/pull/14699#issuecomment-2900515377 This PR does not have an entry in lucene/CHANGES.txt. Consider adding one. If the PR doesn't need a changelog entry, then add the skip-changelog-check label to it and you wil

Re: [PR] Add Query for reranking KnnFloatVectorQuery with full-precision vectors [lucene]

2025-05-22 Thread via GitHub
vigyasharma commented on PR #14009: URL: https://github.com/apache/lucene/pull/14009#issuecomment-2900200570 @dungba88 : I shared an [alternate implementation](https://github.com/apache/lucene/pull/14698) for RerankVectorQuery that I feel can generalize more broadly. **Main Differen

[PR] An alternate implementation for RerankVectorQuery [lucene]

2025-05-22 Thread via GitHub
vigyasharma opened a new pull request, #14698: URL: https://github.com/apache/lucene/pull/14698 This PR presents an alternate to #14009 for `RerankVectorQuery`. It takes an input query, a target vector and a field to use for vector values, and rescores the output of wrapped query with full