Re: [PR] Merge related HashMaps in FieldInfos#FieldNumbers into one map [lucene]

2024-06-06 Thread via GitHub
iverase commented on PR #13460: URL: https://github.com/apache/lucene/pull/13460#issuecomment-2154127830 This class seems to have change quite a bit and some methods did not have a clear path forward for backportso I set the milestone for this change to 10.0.0 -- This is an automated mess

Re: [PR] Improve Test Coverage: added tests for IntRange [lucene]

2024-06-06 Thread via GitHub
harenlin closed pull request #13418: Improve Test Coverage: added tests for IntRange URL: https://github.com/apache/lucene/pull/13418 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific com

Re: [PR] Delete all live docs when query matched a whole segment. [lucene]

2024-06-06 Thread via GitHub
github-actions[bot] commented on PR #13395: URL: https://github.com/apache/lucene/pull/13395#issuecomment-2153622179 This PR has not had activity in the past 2 weeks, labeling it as stale. If the PR is waiting for review, notify the d...@lucene.apache.org list. Thank you for your contributi

Re: [I] CorruptIndexException: docs out of order in merge thread [lucene]

2024-06-06 Thread via GitHub
jpountz commented on issue #13457: URL: https://github.com/apache/lucene/issues/13457#issuecomment-2153269421 The fact that the index is small rules out an overflow I suppose. I'm not familiar with this flavor of the JDK, are you able to test with another vendor to see if the problem still

[PR] Add int8_hnsw backcompat index creawtion to dev tools scripts [lucene]

2024-06-06 Thread via GitHub
benwtrent opened a new pull request, #13465: URL: https://github.com/apache/lucene/pull/13465 I noticed during the 9.11 release, that I failed to update the `addBackcompatIndexes` script when I added new backcompat tests & indices. This commit remedies this. I will backport to 9x

Re: [PR] MultiTermQuery return null for ScoreSupplier [lucene]

2024-06-06 Thread via GitHub
mayya-sharipova merged PR #13454: URL: https://github.com/apache/lucene/pull/13454 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lu

Re: [PR] MultiTermQuery return null for ScoreSupplier [lucene]

2024-06-06 Thread via GitHub
mayya-sharipova commented on code in PR #13454: URL: https://github.com/apache/lucene/pull/13454#discussion_r1630030925 ## lucene/CHANGES.txt: ## @@ -249,7 +249,7 @@ Improvements Optimizations - -(No changes) +* GITHUB#13454: MultiTermQuery return null fo

Re: [PR] Add a MemorySegment Vector scorer - for scoring without copying on-heap [lucene]

2024-06-06 Thread via GitHub
gautamworah96 commented on PR #13339: URL: https://github.com/apache/lucene/pull/13339#issuecomment-2153128147 Hey @ChrisHegarty do we have an open issue for extending this to float vectors? If not, I can create one -- This is an automated message from the Apache Git Service. To respo

[PR] Reduce the heap use of BKDReader instances [lucene]

2024-06-06 Thread via GitHub
original-brownbear opened a new pull request, #13464: URL: https://github.com/apache/lucene/pull/13464 We consume a lot of memory for the `indexIn` slices. If `indexIn` is of type `MemorySegmentIndexInput` the overhead of keeping loads of slices around just for cloning is far higher than th

Re: [PR] MultiTermQuery return null for ScoreSupplier [lucene]

2024-06-06 Thread via GitHub
gsmiller commented on code in PR #13454: URL: https://github.com/apache/lucene/pull/13454#discussion_r1629951445 ## lucene/core/src/java/org/apache/lucene/search/AbstractMultiTermQueryConstantScoreWrapper.java: ## @@ -162,21 +163,29 @@ private WeightOrDocIdSetIterator rewrite(L

Re: [PR] Merge related HashMaps in FieldInfos#FieldNumbers into one map [lucene]

2024-06-06 Thread via GitHub
iverase merged PR #13460: URL: https://github.com/apache/lucene/pull/13460 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucene.apa

Re: [I] Should FieldInfo#FieldNumbers hold one map with index properties instead of a map for each property? [lucene]

2024-06-06 Thread via GitHub
iverase closed issue #13459: Should FieldInfo#FieldNumbers hold one map with index properties instead of a map for each property? URL: https://github.com/apache/lucene/issues/13459 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

Re: [I] Backport gh workflow cleanups to branch_9x [lucene]

2024-06-06 Thread via GitHub
dweiss closed issue #13461: Backport gh workflow cleanups to branch_9x URL: https://github.com/apache/lucene/issues/13461 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To uns

Re: [I] What does the Lucene community think about dimensionality reduction for vectors, and should it be something the library does internally (at merge time perhaps)? [lucene]

2024-06-06 Thread via GitHub
msokolov commented on issue #13403: URL: https://github.com/apache/lucene/issues/13403#issuecomment-2152698409 I've been thinking about PCA a bit and although it can be very useful in some settings, I'm not convinced it really belongs in Lucene, but should more likely be part of a pre-index

[PR] Fix global score update bug in MultiLeafKnnCollector [lucene]

2024-06-06 Thread via GitHub
gsmiller opened a new pull request, #13463: URL: https://github.com/apache/lucene/pull/13463 Addresses the bug described in GH#13462 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific c

Re: [PR] Removed Scorer#getWeight [lucene]

2024-06-06 Thread via GitHub
jpountz commented on PR #13440: URL: https://github.com/apache/lucene/pull/13440#issuecomment-2152626078 Thank you! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsub

Re: [I] Remove Scorer#getWeight. [lucene]

2024-06-06 Thread via GitHub
jpountz closed issue #13410: Remove Scorer#getWeight. URL: https://github.com/apache/lucene/issues/13410 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [PR] Removed Scorer#getWeight [lucene]

2024-06-06 Thread via GitHub
jpountz merged PR #13440: URL: https://github.com/apache/lucene/pull/13440 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucene.apa

Re: [PR] Add test for ghost fields to BaseKnnVectorQueryTestCase. [lucene]

2024-06-06 Thread via GitHub
jpountz merged PR #13455: URL: https://github.com/apache/lucene/pull/13455 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucene.apa

[I] Bug in MultiLeafKnnCollector causes #minCompetitiveSimilarity to stay artificially low in some situations [lucene]

2024-06-06 Thread via GitHub
gsmiller opened a new issue, #13462: URL: https://github.com/apache/lucene/issues/13462 ### Description The logic in MultiLeafKnnCollector for updating the global scores (coordinating across leaves) incorrectly assumes heaps are fully ordered and can fail to add some scores to the gl

Re: [I] Bug in MultiLeafKnnCollector causes #minCompetitiveSimilarity to stay artificially low in some situations [lucene]

2024-06-06 Thread via GitHub
gsmiller commented on issue #13462: URL: https://github.com/apache/lucene/issues/13462#issuecomment-2152618923 I'll propose a fix shortly. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the spec

[I] Backport gh workflow cleanups to branch_9x [lucene]

2024-06-06 Thread via GitHub
dweiss opened a new issue, #13461: URL: https://github.com/apache/lucene/issues/13461 ### Description Synchronize changes with main. ### Version and environment details _No response_ -- This is an automated message from the Apache Git Service. To respond to the message

Re: [PR] Add a github workflow that checks common (and less common) gradle tasks when gradle version is changed [lucene]

2024-06-06 Thread via GitHub
dweiss merged PR #13456: URL: https://github.com/apache/lucene/pull/13456 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucene.apac

Re: [PR] Merge related HashMaps in FieldInfos#FieldNumbers into one map [lucene]

2024-06-06 Thread via GitHub
jpountz commented on code in PR #13460: URL: https://github.com/apache/lucene/pull/13460#discussion_r1629497818 ## lucene/core/src/java/org/apache/lucene/index/FieldInfos.java: ## @@ -664,14 +643,13 @@ synchronized void verifyOrCreateDvOnlyField( * {@code dvType} retur

Re: [PR] Merge related HashMaps in FieldInfos#FieldNumbers into one map [lucene]

2024-06-06 Thread via GitHub
jpountz commented on code in PR #13460: URL: https://github.com/apache/lucene/pull/13460#discussion_r1629485819 ## lucene/core/src/java/org/apache/lucene/index/FieldInfos.java: ## @@ -403,11 +385,8 @@ static final class FieldNumbers { FieldNumbers(String softDeletesFieldNam

Re: [PR] Merge related HashMaps in FieldInfos#FieldNumbers into one map [lucene]

2024-06-06 Thread via GitHub
jpountz commented on code in PR #13460: URL: https://github.com/apache/lucene/pull/13460#discussion_r1629484661 ## lucene/core/src/java/org/apache/lucene/index/FieldInfos.java: ## @@ -403,11 +385,8 @@ static final class FieldNumbers { FieldNumbers(String softDeletesFieldNam

Re: [PR] Merge related HashMaps in FieldInfos#FieldNumbers into one map [lucene]

2024-06-06 Thread via GitHub
iverase commented on code in PR #13460: URL: https://github.com/apache/lucene/pull/13460#discussion_r1629399538 ## lucene/core/src/java/org/apache/lucene/index/FieldInfos.java: ## @@ -664,14 +643,13 @@ synchronized void verifyOrCreateDvOnlyField( * {@code dvType} retur

Re: [PR] Merge related HashMaps in FieldInfos#FieldNumbers into one map [lucene]

2024-06-06 Thread via GitHub
iverase commented on code in PR #13460: URL: https://github.com/apache/lucene/pull/13460#discussion_r1629388801 ## lucene/core/src/java/org/apache/lucene/index/FieldInfos.java: ## @@ -403,11 +385,8 @@ static final class FieldNumbers { FieldNumbers(String softDeletesFieldNam

Re: [PR] Merge related HashMaps in FieldInfos#FieldNumbers into one map [lucene]

2024-06-06 Thread via GitHub
jpountz commented on code in PR #13460: URL: https://github.com/apache/lucene/pull/13460#discussion_r1629348007 ## lucene/core/src/java/org/apache/lucene/index/FieldInfos.java: ## @@ -403,11 +385,8 @@ static final class FieldNumbers { FieldNumbers(String softDeletesFieldNam

Re: [PR] Sparse index: optional skip list on top of doc values [lucene]

2024-06-06 Thread via GitHub
iverase commented on code in PR #13449: URL: https://github.com/apache/lucene/pull/13449#discussion_r1629301507 ## lucene/core/src/java/org/apache/lucene/index/FieldInfo.java: ## @@ -289,6 +302,24 @@ static void verifySameDocValuesType( } } + /** + * Verify that the

Re: [PR] Merge related HashMaps in FieldInfos#FieldNumbers into one map [lucene]

2024-06-06 Thread via GitHub
iverase commented on code in PR #13460: URL: https://github.com/apache/lucene/pull/13460#discussion_r1629301169 ## lucene/core/src/java/org/apache/lucene/index/FieldInfos.java: ## @@ -403,11 +385,8 @@ static final class FieldNumbers { FieldNumbers(String softDeletesFieldNam

Re: [PR] Merge related HashMaps in FieldInfos into one map [lucene]

2024-06-06 Thread via GitHub
iverase commented on code in PR #13460: URL: https://github.com/apache/lucene/pull/13460#discussion_r1629276194 ## lucene/core/src/java/org/apache/lucene/index/FieldInfos.java: ## @@ -701,9 +684,7 @@ synchronized Set getFieldNames() { synchronized void clear() { numb

Re: [PR] Merge related HashMaps in FieldInfos into one map [lucene]

2024-06-06 Thread via GitHub
iverase commented on code in PR #13460: URL: https://github.com/apache/lucene/pull/13460#discussion_r1629275099 ## lucene/core/src/java/org/apache/lucene/index/FieldInfos.java: ## @@ -403,11 +385,8 @@ static final class FieldNumbers { FieldNumbers(String softDeletesFieldNam

Re: [PR] Add a github workflow that checks common (and less common) gradle tasks when gradle version is changed [lucene]

2024-06-06 Thread via GitHub
dweiss commented on PR #13456: URL: https://github.com/apache/lucene/pull/13456#issuecomment-2151966136 I think this is ready and works reasonably well - it caught #13458 already. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

[I] Should FieldInfo hold one map with index properties instead of a map for each property? [lucene]

2024-06-06 Thread via GitHub
iverase opened a new issue, #13459: URL: https://github.com/apache/lucene/issues/13459 We currently have in FieldInfos the following maps: ``` this.indexOptions = new HashMap<>(); this.docValuesType = new HashMap<>(); this.dimensions = new HashMap<>();

Re: [PR] Add a github workflow that checks common (and less common) gradle tasks when gradle version is changed [lucene]

2024-06-06 Thread via GitHub
dweiss commented on PR #13456: URL: https://github.com/apache/lucene/pull/13456#issuecomment-2151961891 I've added a matrix configuration so that one job runs with RUNTIME_JAVA_HOME and the other without it. Seems to be working and increases test coverage. ![image](https://github.co

Re: [PR] Add a github workflow that checks common (and less common) gradle tasks when gradle version is changed [lucene]

2024-06-06 Thread via GitHub
dweiss commented on code in PR #13456: URL: https://github.com/apache/lucene/pull/13456#discussion_r1629249452 ## .github/workflows/run-checks-gradle-upgrade.yml: ## @@ -0,0 +1,88 @@ +name: "Run checks: gradle upgrade" + +on: + workflow_dispatch: + + pull_request: +branche

Re: [PR] Add a github workflow that checks common (and less common) gradle tasks when gradle version is changed [lucene]

2024-06-06 Thread via GitHub
dweiss closed pull request #13456: Add a github workflow that checks common (and less common) gradle tasks when gradle version is changed URL: https://github.com/apache/lucene/pull/13456 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to G

Re: [PR] Add a github workflow that checks common (and less common) gradle tasks when gradle version is changed [lucene]

2024-06-06 Thread via GitHub
dweiss commented on PR #13456: URL: https://github.com/apache/lucene/pull/13456#issuecomment-2151924570 Let me close this PR so that it doesn't rerun all the checks while I'm trying to figure it out locally. -- This is an automated message from the Apache Git Service. To respond to the me

Re: [I] Update WrapperDownloader to accept java 22 and correct deprecated new URL API [lucene]

2024-06-06 Thread via GitHub
dweiss closed issue #13458: Update WrapperDownloader to accept java 22 and correct deprecated new URL API URL: https://github.com/apache/lucene/issues/13458 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[I] Update WrapperDownloader to accept java 22 and correct deprecated new URL API [lucene]

2024-06-06 Thread via GitHub
dweiss opened a new issue, #13458: URL: https://github.com/apache/lucene/issues/13458 ### Description A follow up to #13453 ### Version and environment details _No response_ -- This is an automated message from the Apache Git Service. To respond to the message, please

Re: [PR] Add a github workflow that checks common (and less common) gradle tasks when gradle version is changed [lucene]

2024-06-06 Thread via GitHub
dweiss commented on PR #13456: URL: https://github.com/apache/lucene/pull/13456#issuecomment-2151807298 > I think that this is good. Would it run on all CI jobs, or you somehow opt-in? There is a filtering restriction - it'll run on PRs and pushes to these two branches and when the c

Re: [PR] Sparse index: optional skip list on top of doc values [lucene]

2024-06-06 Thread via GitHub
easyice commented on code in PR #13449: URL: https://github.com/apache/lucene/pull/13449#discussion_r1629013375 ## lucene/core/src/java/org/apache/lucene/index/FieldInfo.java: ## @@ -289,6 +302,24 @@ static void verifySameDocValuesType( } } + /** + * Verify that the

Re: [PR] Sparse index: optional skip list on top of doc values [lucene]

2024-06-06 Thread via GitHub
iverase commented on code in PR #13449: URL: https://github.com/apache/lucene/pull/13449#discussion_r1629012824 ## lucene/core/src/java/org/apache/lucene/codecs/lucene90/Lucene90DocValuesProducer.java: ## @@ -197,6 +215,17 @@ private NumericEntry readNumeric(IndexInput meta) thr

Re: [PR] Sparse index: optional skip list on top of doc values [lucene]

2024-06-06 Thread via GitHub
iverase commented on code in PR #13449: URL: https://github.com/apache/lucene/pull/13449#discussion_r1628996406 ## lucene/core/src/java/org/apache/lucene/codecs/lucene90/Lucene90DocValuesProducer.java: ## @@ -1749,4 +1781,88 @@ long getLongValue(long index) throws IOException {

Re: [PR] Add a github workflow that checks common (and less common) gradle tasks when gradle version is changed [lucene]

2024-06-06 Thread via GitHub
ChrisHegarty commented on PR #13456: URL: https://github.com/apache/lucene/pull/13456#issuecomment-2151641099 I think that this is good. Would it run on all CI jobs, or you somehow opt-in? Running with `RUNTIME_JAVA_HOME` does have a slightly different code path in the gradle buil

Re: [PR] Update Gradle wrapper to 8.8 - supports Java 22 [lucene]

2024-06-06 Thread via GitHub
ChrisHegarty merged PR #13453: URL: https://github.com/apache/lucene/pull/13453 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucen

Re: [PR] Update Gradle wrapper to 8.8 - supports Java 22 [lucene]

2024-06-06 Thread via GitHub
dweiss commented on PR #13453: URL: https://github.com/apache/lucene/pull/13453#issuecomment-2151610526 > Yeah, I think the biggest problem is to find one task which executes everything in the whole Gradle Build... We would need some Gradle Coverage Report! I don't think we can find