Re: [PR] Give a hint to `IndexInput` about slices that have a forward-only access pattern. [lucene]

2024-06-05 Thread via GitHub
jpountz commented on code in PR #13450: URL: https://github.com/apache/lucene/pull/13450#discussion_r1627103936 ## lucene/core/src/java21/org/apache/lucene/store/MemorySegmentIndexInput.java: ## @@ -370,6 +370,16 @@ public void prefetch(long offset, long length) throws IOExcept

Re: [PR] Fix IndexOutOfBoundsException thrown in DefaultPassageFormatter by unordered matches [lucene]

2024-06-05 Thread via GitHub
scampi commented on PR #13315: URL: https://github.com/apache/lucene/pull/13315#issuecomment-2149085180 @mikemccand @romseygeek thanks for the review! > What a fun and tricky corner case -- thank you @scampi for uncovering this, showing the bug with the added unit tests, and the tenta

Re: [PR] Sparse index: optional skip list on top of doc values [lucene]

2024-06-05 Thread via GitHub
ChrisHegarty commented on code in PR #13449: URL: https://github.com/apache/lucene/pull/13449#discussion_r1626571087 ## lucene/core/src/java/org/apache/lucene/index/DocValuesSkipper.java: ## @@ -0,0 +1,95 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or m

Re: [I] org.apache.lucene.index.IndexFormatTooNewException on arm64 [lucene]

2024-06-05 Thread via GitHub
easyice commented on issue #13452: URL: https://github.com/apache/lucene/issues/13452#issuecomment-2149099127 It looks like you are using a lower version of lucene to read the index generated by a higher version. Are you sure the version with this exception is lucene 9.10? -- This is an

Re: [PR] Sparse index: optional skip list on top of doc values [lucene]

2024-06-05 Thread via GitHub
iverase commented on code in PR #13449: URL: https://github.com/apache/lucene/pull/13449#discussion_r1627219921 ## lucene/core/src/java/org/apache/lucene/codecs/lucene90/Lucene90DocValuesProducer.java: ## @@ -1690,4 +1722,78 @@ long getLongValue(long index) throws IOException {

Re: [PR] Sparse index: optional skip list on top of doc values [lucene]

2024-06-05 Thread via GitHub
iverase commented on code in PR #13449: URL: https://github.com/apache/lucene/pull/13449#discussion_r1627219039 ## lucene/core/src/java/org/apache/lucene/index/DocValuesSkipper.java: ## @@ -0,0 +1,95 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more +

Re: [PR] Sparse index: optional skip list on top of doc values [lucene]

2024-06-05 Thread via GitHub
jpountz commented on code in PR #13449: URL: https://github.com/apache/lucene/pull/13449#discussion_r1627225989 ## lucene/core/src/java/org/apache/lucene/codecs/lucene90/Lucene90DocValuesProducer.java: ## @@ -1690,4 +1722,78 @@ long getLongValue(long index) throws IOException {

Re: [I] org.apache.lucene.index.IndexFormatTooNewException on arm64 [lucene]

2024-06-05 Thread via GitHub
suddendust commented on issue #13452: URL: https://github.com/apache/lucene/issues/13452#issuecomment-2149197649 https://github.com/apache/lucene/assets/84911643/c4bd14fe-cbb3-464b-998d-25c224d6becc";> Yes it is 9.10.0. -- This is an automated message from the Apache Git Service. T

[PR] Update Gradle wrapper to 8.8 - supports Java 22 [lucene]

2024-06-05 Thread via GitHub
ChrisHegarty opened a new pull request, #13453: URL: https://github.com/apache/lucene/pull/13453 This commit updates the Gradle wrapper to 8.8, which has support for Java 22. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

Re: [PR] Update Gradle wrapper to 8.8 - supports Java 22 [lucene]

2024-06-05 Thread via GitHub
ChrisHegarty commented on code in PR #13453: URL: https://github.com/apache/lucene/pull/13453#discussion_r1627277388 ## gradle/testing/alternative-jdk-support.gradle: ## @@ -88,6 +88,6 @@ if (jvmGradle != jvmCurrent) { // Set up root project's properties. rootProject.ext.runti

Re: [I] Improve Lucene's I/O concurrency [lucene]

2024-06-05 Thread via GitHub
jpountz commented on issue #13179: URL: https://github.com/apache/lucene/issues/13179#issuecomment-2149257220 > Then before evaluating if these docs matches TwoPhaseIterator or not, we can perform prefetch on these buffered docs (via some prepareMatches mechanism on TwoPhaseIterator).

Re: [PR] Update Gradle wrapper to 8.8 - supports Java 22 [lucene]

2024-06-05 Thread via GitHub
ChrisHegarty commented on PR #13453: URL: https://github.com/apache/lucene/pull/13453#issuecomment-2149264617 While not strictly necessary to be able to run tests with JDK 22, upgrading the gradle wrapper will simplify usage. Since one can just set JAVA_HOME, rather than both JAVA_HOME and

Re: [PR] Update Gradle wrapper to 8.8 - supports Java 22 [lucene]

2024-06-05 Thread via GitHub
ChrisHegarty commented on PR #13453: URL: https://github.com/apache/lucene/pull/13453#issuecomment-2149287115 @uschindler I was motivated to do this upgrade since stumbling into issues trying to verify the 9.11 RC1 build with `--test-alternative-java `, and kept running into silly configura

Re: [I] AnalyzingSuggester exception because of length restriction: java.lang.IllegalArgumentException: len must be <= 32767; got 38751 [LUCENE-6012] [lucene]

2024-06-05 Thread via GitHub
msokolov commented on issue #7074: URL: https://github.com/apache/lucene/issues/7074#issuecomment-2149357993 Would it be possible and desirable to truncate the input rather than failing with an error? Again though I would echo the questions around what you expect to happen in a suggester wi

Re: [PR] Sparse index: optional skip list on top of doc values [lucene]

2024-06-05 Thread via GitHub
ChrisHegarty commented on code in PR #13449: URL: https://github.com/apache/lucene/pull/13449#discussion_r1627434696 ## lucene/core/src/java/org/apache/lucene/codecs/lucene90/Lucene90DocValuesProducer.java: ## @@ -1690,4 +1722,78 @@ long getLongValue(long index) throws IOExcepti

Re: [PR] Sparse index: optional skip list on top of doc values [lucene]

2024-06-05 Thread via GitHub
ChrisHegarty commented on code in PR #13449: URL: https://github.com/apache/lucene/pull/13449#discussion_r1627440236 ## lucene/core/src/java/org/apache/lucene/index/DocValuesSkipper.java: ## @@ -0,0 +1,101 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or

Re: [PR] Sparse index: optional skip list on top of doc values [lucene]

2024-06-05 Thread via GitHub
gf2121 commented on code in PR #13449: URL: https://github.com/apache/lucene/pull/13449#discussion_r1627521914 ## lucene/core/src/java/org/apache/lucene/codecs/lucene90/Lucene90DocValuesProducer.java: ## @@ -1690,4 +1722,78 @@ long getLongValue(long index) throws IOException {

Re: [PR] Set `ReadAdvice#NORMAL` on files that have a forward-only access pattern. [lucene]

2024-06-05 Thread via GitHub
jpountz merged PR #13450: URL: https://github.com/apache/lucene/pull/13450 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucene.apa

Re: [PR] Add prefetching for doc values and norms. [lucene]

2024-06-05 Thread via GitHub
jpountz merged PR #13411: URL: https://github.com/apache/lucene/pull/13411 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucene.apa

Re: [PR] Sparse index: optional skip list on top of doc values [lucene]

2024-06-05 Thread via GitHub
iverase commented on code in PR #13449: URL: https://github.com/apache/lucene/pull/13449#discussion_r1627604017 ## lucene/core/src/java/org/apache/lucene/codecs/lucene90/Lucene90DocValuesProducer.java: ## @@ -1690,4 +1722,78 @@ long getLongValue(long index) throws IOException {

Re: [PR] Sparse index: optional skip list on top of doc values [lucene]

2024-06-05 Thread via GitHub
iverase commented on code in PR #13449: URL: https://github.com/apache/lucene/pull/13449#discussion_r1627603577 ## lucene/core/src/java/org/apache/lucene/index/DocValuesSkipper.java: ## @@ -0,0 +1,101 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more

Re: [PR] Update Gradle wrapper to 8.8 - supports Java 22 [lucene]

2024-06-05 Thread via GitHub
ChrisHegarty commented on PR #13453: URL: https://github.com/apache/lucene/pull/13453#issuecomment-2149663385 I'm planning on backporting this to the 9.x branch, unless there are concerns. -- This is an automated message from the Apache Git Service. To respond to the message, please log o

Re: [PR] Update Gradle wrapper to 8.8 - supports Java 22 [lucene]

2024-06-05 Thread via GitHub
ChrisHegarty commented on PR #13453: URL: https://github.com/apache/lucene/pull/13453#issuecomment-2149664517 Thanks for the review @dweiss. ❤️ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to th

Re: [PR] Update Gradle wrapper to 8.8 - supports Java 22 [lucene]

2024-06-05 Thread via GitHub
dweiss commented on PR #13453: URL: https://github.com/apache/lucene/pull/13453#issuecomment-2149701905 You're very welcome. Upgrading gradle is risky as it may entail changes in their APIs that only surface once a task is actually used (dynamic code) but there's really no other way to do i

Re: [PR] Deprecate COSINE VectorSimilarity function [lucene]

2024-06-05 Thread via GitHub
benwtrent commented on PR #13308: URL: https://github.com/apache/lucene/pull/13308#issuecomment-2149920192 @Pulkitg64 sorry for radio silence, I was out of office and swamped with other things. If you could move the change log to 9.12, I can merge and backport. -- This is an automated m

Re: [PR] Implement Weight#count for vector values in the FieldExistsQuery [lucene]

2024-06-05 Thread via GitHub
benwtrent commented on PR #13322: URL: https://github.com/apache/lucene/pull/13322#issuecomment-2149930924 @bugmakerr Once you handle the conflicts, I can merge and backport. I think this is ready to go unless @jpountz has any outstanding concerns. -- This is an automated message from

Re: [PR] Add BitVectors format and make flat vectors format easier to extend [lucene]

2024-06-05 Thread via GitHub
benwtrent commented on PR #13288: URL: https://github.com/apache/lucene/pull/13288#issuecomment-2149954395 @navneet1v > Does this mean now Lucene supports BitVectorsFormat officially? The answer is no. > Or it was more a prototype and not intended for production use?

[PR] MultiTermQuery return null for ScoreSupplier [lucene]

2024-06-05 Thread via GitHub
mayya-sharipova opened a new pull request, #13454: URL: https://github.com/apache/lucene/pull/13454 MultiTermQuery return null for ScoreSupplier if there are no terms in an index that match query terms. With the introduction of PR #12156 we saw degradation in performance of bool

Re: [PR] MultiTermQuery return null for ScoreSupplier [lucene]

2024-06-05 Thread via GitHub
mayya-sharipova commented on PR #13454: URL: https://github.com/apache/lucene/pull/13454#issuecomment-2150107425 @gsmiller @jpountz I don't know all the implications for this, so your feedback is welcome. -- This is an automated message from the Apache Git Service. To respond to the mes

Re: [PR] Implement Weight#count for vector values in the FieldExistsQuery [lucene]

2024-06-05 Thread via GitHub
bugmakerr commented on PR #13322: URL: https://github.com/apache/lucene/pull/13322#issuecomment-2150212641 @benwtrent done -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment

Re: [I] Improve AbstractMultiTermQueryConstantScoreWrapper#RewritingWeight ScorerSupplier cost estimation [lucene]

2024-06-05 Thread via GitHub
rquesada-tibco commented on issue #13029: URL: https://github.com/apache/lucene/issues/13029#issuecomment-2150258368 fyi @jboeuf-tibco, @fledigol-tibco -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the U

Re: [PR] Update Gradle wrapper to 8.8 - supports Java 22 [lucene]

2024-06-05 Thread via GitHub
uschindler commented on PR #13453: URL: https://github.com/apache/lucene/pull/13453#issuecomment-2150292708 > @uschindler I was motivated to do this upgrade since stumbling into issues trying to verify the 9.11 RC1 build with `--test-alternative-java `, and kept running into silly configura

Re: [PR] Update Gradle wrapper to 8.8 - supports Java 22 [lucene]

2024-06-05 Thread via GitHub
uschindler commented on PR #13453: URL: https://github.com/apache/lucene/pull/13453#issuecomment-2150296383 The problem with gradle updates is always: theres is always a small piece of build logic not tested that breaks. Its hard to test the whole thing unless you also run regenerate or do

Re: [PR] MultiTermQuery return null for ScoreSupplier [lucene]

2024-06-05 Thread via GitHub
jpountz commented on code in PR #13454: URL: https://github.com/apache/lucene/pull/13454#discussion_r1627942880 ## lucene/core/src/java/org/apache/lucene/search/AbstractMultiTermQueryConstantScoreWrapper.java: ## @@ -232,12 +221,24 @@ public ScorerSupplier scorerSupplier(LeafRea

[PR] Add test for ghost fields to BaseKnnVectorQueryTestCase. [lucene]

2024-06-05 Thread via GitHub
jpountz opened a new pull request, #13455: URL: https://github.com/apache/lucene/pull/13455 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-m

Re: [PR] Implement Weight#count for vector values in the FieldExistsQuery [lucene]

2024-06-05 Thread via GitHub
jpountz commented on code in PR #13322: URL: https://github.com/apache/lucene/pull/13322#discussion_r1627993328 ## lucene/core/src/java/org/apache/lucene/search/FieldExistsQuery.java: ## @@ -277,4 +276,20 @@ private String buildErrorMsg(FieldInfo fieldInfo) { + fieldInf

Re: [PR] Update Gradle wrapper to 8.8 - supports Java 22 [lucene]

2024-06-05 Thread via GitHub
ChrisHegarty commented on PR #13453: URL: https://github.com/apache/lucene/pull/13453#issuecomment-2150435190 I ran regenerate - it found another small issue with InstallationLocation, now fixed. -- This is an automated message from the Apache Git Service. To respond to the message, pleas

Re: [PR] Update Gradle wrapper to 8.8 - supports Java 22 [lucene]

2024-06-05 Thread via GitHub
uschindler commented on PR #13453: URL: https://github.com/apache/lucene/pull/13453#issuecomment-2150671839 > I ran regenerate - it found another small issue with InstallationLocation, now fixed. Yeah, I think the biggest problem is to find one task which executes everything in the w

Re: [PR] MultiTermQuery return null for ScoreSupplier [lucene]

2024-06-05 Thread via GitHub
mayya-sharipova commented on code in PR #13454: URL: https://github.com/apache/lucene/pull/13454#discussion_r1628251924 ## lucene/core/src/java/org/apache/lucene/search/AbstractMultiTermQueryConstantScoreWrapper.java: ## @@ -153,39 +153,28 @@ protected abstract WeightOrDocIdSetI

Re: [PR] MultiTermQuery return null for ScoreSupplier [lucene]

2024-06-05 Thread via GitHub
mayya-sharipova commented on code in PR #13454: URL: https://github.com/apache/lucene/pull/13454#discussion_r1628252725 ## lucene/core/src/java/org/apache/lucene/search/AbstractMultiTermQueryConstantScoreWrapper.java: ## @@ -232,12 +221,24 @@ public ScorerSupplier scorerSupplier

Re: [PR] MultiTermQuery return null for ScoreSupplier [lucene]

2024-06-05 Thread via GitHub
mayya-sharipova commented on PR #13454: URL: https://github.com/apache/lucene/pull/13454#issuecomment-2150704539 @jpountz Thanks for your feedback, I addressed it, and the code looks much better with your suggestions! -- This is an automated message from the Apache Git Service. To res

Re: [PR] Implement Weight#count for vector values in the FieldExistsQuery [lucene]

2024-06-05 Thread via GitHub
benwtrent merged PR #13322: URL: https://github.com/apache/lucene/pull/13322 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucene.a

[PR] Add a github workflow that checks common (and less common) gradle tasks when gradle version is changed [lucene]

2024-06-05 Thread via GitHub
dweiss opened a new pull request, #13456: URL: https://github.com/apache/lucene/pull/13456 This is mostly an automated sanity check that common (and less common) tasks work after an upgrade. -- This is an automated message from the Apache Git Service. To respond to the message, please log

Re: [PR] Add a github workflow that checks common (and less common) gradle tasks when gradle version is changed [lucene]

2024-06-05 Thread via GitHub
dweiss commented on PR #13456: URL: https://github.com/apache/lucene/pull/13456#issuecomment-2150773147 This is a follow up to #13453 - perhaps it'll make life a bit easier. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

Re: [PR] Sparse index: optional skip list on top of doc values [lucene]

2024-06-05 Thread via GitHub
benwtrent commented on code in PR #13449: URL: https://github.com/apache/lucene/pull/13449#discussion_r1628362719 ## lucene/core/src/java/org/apache/lucene/codecs/lucene90/Lucene90DocValuesProducer.java: ## @@ -1749,4 +1781,88 @@ long getLongValue(long index) throws IOException

Re: [I] AnalyzingSuggester exception because of length restriction: java.lang.IllegalArgumentException: len must be <= 32767; got 38751 [LUCENE-6012] [lucene]

2024-06-05 Thread via GitHub
dmaziuk commented on issue #7074: URL: https://github.com/apache/lucene/issues/7074#issuecomment-2151247593 After playing some more with it, I'd say the problem is in The Fine Manual. The only clear concise explanation of what the suggester actually does that my google-fu managed to turn up

Re: [I] Improve Lucene's I/O concurrency [lucene]

2024-06-05 Thread via GitHub
sohami commented on issue #13179: URL: https://github.com/apache/lucene/issues/13179#issuecomment-2151313696 > > Then before evaluating if these docs matches TwoPhaseIterator or not, we can perform prefetch on these buffered docs (via some prepareMatches mechanism on TwoPhaseIterator). >

[I] CorruptIndexException: docs out of order in merge thread [lucene]

2024-06-05 Thread via GitHub
swapnilsvaidya opened a new issue, #13457: URL: https://github.com/apache/lucene/issues/13457 ### Description We are using OpenSearch 1.2.3 to index our data. We are observing the following CorruptIndexException frequently --