Re: [PR] LUCENE-8739: custom codec providing Zstandard compression/decompression [lucene]

2024-03-22 Thread via GitHub
github-actions[bot] commented on PR #439: URL: https://github.com/apache/lucene/pull/439#issuecomment-2016214504 This PR has not had activity in the past 2 weeks, labeling it as stale. If the PR is waiting for review, notify the d...@lucene.apache.org list. Thank you for your contribution!

Re: [PR] Add timeout support to AbstractKnnVectorQuery [lucene]

2024-03-22 Thread via GitHub
kaivalnp commented on PR #13202: URL: https://github.com/apache/lucene/pull/13202#issuecomment-2016087745 ### baseline with `null` timeout ``` 0.965 1.10 100 50 16 100 16750 1.00 post-filter 0.992 2.82 100 400 16 10

[PR] Add timeout support to AbstractKnnVectorQuery [lucene]

2024-03-22 Thread via GitHub
kaivalnp opened a new pull request, #13202: URL: https://github.com/apache/lucene/pull/13202 ### Description #927 added timeout support to [`IndexSearcher#search`](https://github.com/apache/lucene/blob/75e1ebc4501733a6a9857858b3fd9c30304234d0/lucene/core/src/java/org/apache/lucene/sea

[PR] Get better cost estimate on MultiTermQuery over few terms [lucene]

2024-03-22 Thread via GitHub
msfroh opened a new pull request, #13201: URL: https://github.com/apache/lucene/pull/13201 ### Description The existing cost estimate for multi-term queries was taking the worst-case scenario (e.g the query matches every term in the field), to provide an upper bound on the co

[PR] Add new pluggable vector similarity to field info [lucene]

2024-03-22 Thread via GitHub
benwtrent opened a new pull request, #13200: URL: https://github.com/apache/lucene/pull/13200 Opening this PR VERY early for discussion as I didn't want to spend a bunch of time doing all the refactoring necessary if we thought the API was wrong. This is nowhere near finished. Th

Re: [PR] Fix TestTaxonomyFacetValueSource.testRandom [lucene]

2024-03-22 Thread via GitHub
iamsanjay commented on PR #13198: URL: https://github.com/apache/lucene/pull/13198#issuecomment-2015519505 @benwtrent I added above code and for some reason the Facet results have completely changed and did not include the children with value zero. Hence, it's passing. However, If i try wit

Re: [PR] DRAFT: Add support for posix_madvise to Java 21 MMapDirectory [lucene]

2024-03-22 Thread via GitHub
uschindler commented on PR #13196: URL: https://github.com/apache/lucene/pull/13196#issuecomment-2015347052 Do we have any benchmarks by Mike? Some on memory restricted devices. An idea would be a limited VM or by using the memory hog in Mike's benchmark. -- This is an automated message f

Re: [PR] DRAFT: Add support for posix_madvise to Java 21 MMapDirectory [lucene]

2024-03-22 Thread via GitHub
uschindler commented on code in PR #13196: URL: https://github.com/apache/lucene/pull/13196#discussion_r1535773522 ## gradle/testing/defaults-tests.gradle: ## @@ -132,6 +132,8 @@ allprojects { if (rootProject.vectorIncubatorJavaVersions.contains(rootProject.runtimeJavaVe

Re: [PR] DRAFT: Add support for posix_madvise to Java 21 MMapDirectory [lucene]

2024-03-22 Thread via GitHub
uschindler commented on PR #13196: URL: https://github.com/apache/lucene/pull/13196#issuecomment-2015341476 > > But when we backport this to 9.x, I tend to add another system property to disable this feature. > > Why is this extra property needed? And not just enable madvise for the

Re: [PR] DRAFT: Add support for posix_madvise to Java 21 MMapDirectory [lucene]

2024-03-22 Thread via GitHub
ChrisHegarty commented on PR #13196: URL: https://github.com/apache/lucene/pull/13196#issuecomment-2015293871 > But when we backport this to 9.x, I tend to add another system property to disable this feature. Why is this extra property needed? And not just enable madvise for the Memo

Re: [PR] DRAFT: Add support for posix_madvise to Java 21 MMapDirectory [lucene]

2024-03-22 Thread via GitHub
ChrisHegarty commented on code in PR #13196: URL: https://github.com/apache/lucene/pull/13196#discussion_r1535725315 ## gradle/testing/defaults-tests.gradle: ## @@ -132,6 +132,8 @@ allprojects { if (rootProject.vectorIncubatorJavaVersions.contains(rootProject.runtimeJava

Re: [PR] New int4 scalar quantization [lucene]

2024-03-22 Thread via GitHub
benwtrent commented on code in PR #13197: URL: https://github.com/apache/lucene/pull/13197#discussion_r1535715050 ## lucene/core/src/java/org/apache/lucene/codecs/lucene99/OffHeapQuantizedHalfByteVectorValues.java: ## @@ -0,0 +1,299 @@ +/* + * Licensed to the Apache Software Fou

Re: [PR] DRAFT: Add support for posix_madvise to Java 21 MMapDirectory [lucene]

2024-03-22 Thread via GitHub
uschindler commented on PR #13196: URL: https://github.com/apache/lucene/pull/13196#issuecomment-2015267061 I added more documentation and improved the following: - the warning prints the correct command line based on how Lucene is used in modularized or non-modularized code - added th

Re: [PR] DRAFT: Add support for posix_madvise to Java 21 MMapDirectory [lucene]

2024-03-22 Thread via GitHub
ChrisHegarty commented on PR #13196: URL: https://github.com/apache/lucene/pull/13196#issuecomment-2015261869 > I agree, here it triggers platform specific madvise(). We may add mocks for testing, but I am not sure if it is worth. Maybe the easiest would be to extend IOContext at a later ti

Re: [PR] New int4 scalar quantization [lucene]

2024-03-22 Thread via GitHub
jpountz commented on code in PR #13197: URL: https://github.com/apache/lucene/pull/13197#discussion_r1535656045 ## lucene/core/src/java/org/apache/lucene/codecs/lucene99/OffHeapQuantizedHalfByteVectorValues.java: ## @@ -0,0 +1,299 @@ +/* + * Licensed to the Apache Software Found

Re: [PR] DRAFT: Add support for posix_madvise to Java 21 MMapDirectory [lucene]

2024-03-22 Thread via GitHub
uschindler commented on PR #13196: URL: https://github.com/apache/lucene/pull/13196#issuecomment-2015178992 I added a basic test that verifies that we have madvise support enabled on Linux and Macos. This also added One question: In Lucene 9.x we had some system properties to disable

Re: [PR] Fix TestTaxonomyFacetValueSource.testRandom [lucene]

2024-03-22 Thread via GitHub
benwtrent commented on PR #13198: URL: https://github.com/apache/lucene/pull/13198#issuecomment-2015144997 FYI, the test is fixed by doing the following instead: ``` --- a/lucene/facet/src/test/org/apache/lucene/facet/taxonomy/TestTaxonomyFacetValueSource.java +++ b/lucene/face

Re: [PR] DRAFT: Add support for posix_madvise to Java 21 MMapDirectory [lucene]

2024-03-22 Thread via GitHub
jpountz commented on PR #13196: URL: https://github.com/apache/lucene/pull/13196#issuecomment-2015086424 I'm curious if you've already thought of how we could update the `IOContext` on our merge instances. If I'm not mistaken, given how reader pooling works in `IndexWriter`, if you use a `S

Re: [PR] DRAFT: Add support for posix_madvise to Java 21 MMapDirectory [lucene]

2024-03-22 Thread via GitHub
uschindler commented on PR #13196: URL: https://github.com/apache/lucene/pull/13196#issuecomment-2015060974 > > ... Additionally, I assume we can have unit tests for NativeAccess directly. Lemme take a quick look at this. > > Dammit! I forgot about how our code is organised. Over in E

Re: [PR] DRAFT: Add support for posix_madvise to Java 21 MMapDirectory [lucene]

2024-03-22 Thread via GitHub
uschindler commented on code in PR #13196: URL: https://github.com/apache/lucene/pull/13196#discussion_r1535535951 ## lucene/core/src/java21/org/apache/lucene/store/PosixNativeAccess.java: ## @@ -0,0 +1,123 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or

Re: [PR] DRAFT: Add support for posix_madvise to Java 21 MMapDirectory [lucene]

2024-03-22 Thread via GitHub
rmuir commented on code in PR #13196: URL: https://github.com/apache/lucene/pull/13196#discussion_r1535491072 ## lucene/core/src/java21/org/apache/lucene/store/PosixNativeAccess.java: ## @@ -0,0 +1,123 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more

Re: [PR] DRAFT: Add support for posix_madvise to Java 21 MMapDirectory [lucene]

2024-03-22 Thread via GitHub
rmuir commented on code in PR #13196: URL: https://github.com/apache/lucene/pull/13196#discussion_r1535488782 ## lucene/core/src/java21/org/apache/lucene/store/PosixNativeAccess.java: ## @@ -0,0 +1,123 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more

Re: [PR] DRAFT: Add support for posix_madvise to Java 21 MMapDirectory [lucene]

2024-03-22 Thread via GitHub
rmuir commented on code in PR #13196: URL: https://github.com/apache/lucene/pull/13196#discussion_r1535485315 ## lucene/core/src/java/org/apache/lucene/store/IOContext.java: ## @@ -124,6 +142,7 @@ public int hashCode() { result = prime * result + ((flushInfo == null) ? 0 :

Re: [PR] DRAFT: Add support for posix_madvise to Java 21 MMapDirectory [lucene]

2024-03-22 Thread via GitHub
ChrisHegarty commented on PR #13196: URL: https://github.com/apache/lucene/pull/13196#issuecomment-2014959683 >... Additionally, I assume we can have unit tests for NativeAccess directly. Lemme take a quick look at this. Dammit! I forgot about how our code is organised. Over in Elasti

Re: [PR] DRAFT: Add support for posix_madvise to Java 21 MMapDirectory [lucene]

2024-03-22 Thread via GitHub
rmuir commented on code in PR #13196: URL: https://github.com/apache/lucene/pull/13196#discussion_r1535484417 ## lucene/core/src/java/org/apache/lucene/store/IOContext.java: ## @@ -44,24 +44,32 @@ public enum Context { /** This flag indicates that the file will be opened, the

Re: [PR] DRAFT: Add support for posix_madvise to Java 21 MMapDirectory [lucene]

2024-03-22 Thread via GitHub
ChrisHegarty commented on PR #13196: URL: https://github.com/apache/lucene/pull/13196#issuecomment-2014949144 > We can decide if we want to backport it (should not be too hard). If yes, I will only backport to Java 21 (not 20 or 19). ++ Java 21 is fine. > I will add some gette

Re: [PR] DRAFT: Add support for posix_madvise to Java 21 MMapDirectory [lucene]

2024-03-22 Thread via GitHub
uschindler commented on PR #13196: URL: https://github.com/apache/lucene/pull/13196#issuecomment-2014920137 It looks like we are close for making this ready to be added to main branch. We can decide if we want to backport it (should not be too hard). If yes, I will only backport to Java 21

Re: [PR] DRAFT: Add support for posix_madvise to Java 21 MMapDirectory [lucene]

2024-03-22 Thread via GitHub
ChrisHegarty commented on PR #13196: URL: https://github.com/apache/lucene/pull/13196#issuecomment-2014875051 >.. > It is always a good idea to run one of those tests with `-Ptests.verbose=true` to see if the INFO mesage about madvise is enabled is printed on console: > > ``` >

Re: [PR] DRAFT: Add support for posix_madvise to Java 21 MMapDirectory [lucene]

2024-03-22 Thread via GitHub
uschindler commented on PR #13196: URL: https://github.com/apache/lucene/pull/13196#issuecomment-2014868082 > > ./gradlew :lucene:core:test -Ptests.directory=MMapDirectory > > All tests pass on my Mac M2. It is always a good idea to run one of those tests with `-Ptests.verbose=

Re: [PR] Fix TestTaxonomyFacetValueSource.testRandom [lucene]

2024-03-22 Thread via GitHub
stefanvodita commented on PR #13198: URL: https://github.com/apache/lucene/pull/13198#issuecomment-2014855713 Thanks for working on this. Once #12966 is merged, solving #12585, we would want to revert this change, right? -- This is an automated message from the Apache Git Service. To resp

Re: [PR] DRAFT: Add support for posix_madvise to Java 21 MMapDirectory [lucene]

2024-03-22 Thread via GitHub
uschindler commented on PR #13196: URL: https://github.com/apache/lucene/pull/13196#issuecomment-2014854516 > > > One thing: Should we remove the NoopNativeAccess class and just add a null check? > > > > > > Null is a little easier to reason about in the code at the use-site, sin

Re: [PR] DRAFT: Add support for posix_madvise to Java 21 MMapDirectory [lucene]

2024-03-22 Thread via GitHub
ChrisHegarty commented on PR #13196: URL: https://github.com/apache/lucene/pull/13196#issuecomment-2014855899 > ./gradlew :lucene:core:test -Ptests.directory=MMapDirectory All tests pass on my Mac M2. Native constants are same as yours. ``` $ xcrun --show-sdk-path /Libr

Re: [PR] DRAFT: Add support for posix_madvise to Java 21 MMapDirectory [lucene]

2024-03-22 Thread via GitHub
uschindler commented on PR #13196: URL: https://github.com/apache/lucene/pull/13196#issuecomment-2014843058 > > One thing: Should we remove the NoopNativeAccess class and just add a null check? > > Null is a little easier to reason about in the code at the use-site, since the reader

Re: [PR] DRAFT: Add support for posix_madvise to Java 21 MMapDirectory [lucene]

2024-03-22 Thread via GitHub
uschindler commented on PR #13196: URL: https://github.com/apache/lucene/pull/13196#issuecomment-2014839549 It would be good if you always run tests with `./gradlew :lucene:core:test -Ptests.directory=MMapDirectory` This will use MMapDir for all tests. I have no macos machine, if some

Re: [PR] DRAFT: Add support for posix_madvise to Java 21 MMapDirectory [lucene]

2024-03-22 Thread via GitHub
ChrisHegarty commented on PR #13196: URL: https://github.com/apache/lucene/pull/13196#issuecomment-2014837581 > One thing: Should we remove the NoopNativeAccess class and just add a null check? Null is a little easier to reason about in the code at the use-site, since the reader is s

Re: [PR] DRAFT: Add support for posix_madvise to Java 21 MMapDirectory [lucene]

2024-03-22 Thread via GitHub
ChrisHegarty commented on PR #13196: URL: https://github.com/apache/lucene/pull/13196#issuecomment-2014833076 I took another run at static final ;-). https://github.com/uschindler/lucene/pull/5 (if we still don't want it, then I'll drop it) -- This is an automated message from the Apache

Re: [PR] DRAFT: Add support for posix_madvise to Java 21 MMapDirectory [lucene]

2024-03-22 Thread via GitHub
uschindler commented on PR #13196: URL: https://github.com/apache/lucene/pull/13196#issuecomment-2014829943 Thanks for the PR contributions. You may also commit directly to this branch, unless the change is a complete rewrite :-) -- This is an automated message from the Apache Git Service

Re: [PR] New int4 scalar quantization [lucene]

2024-03-22 Thread via GitHub
tteofili commented on PR #13197: URL: https://github.com/apache/lucene/pull/13197#issuecomment-2014822545 @benwtrent to me it makes sense to have quantization `bits` configurable in this case. -- This is an automated message from the Apache Git Service. To respond to the message, plea

Re: [PR] DRAFT: Add support for posix_madvise to Java 21 MMapDirectory [lucene]

2024-03-22 Thread via GitHub
uschindler commented on code in PR #13196: URL: https://github.com/apache/lucene/pull/13196#discussion_r1535372859 ## lucene/core/src/java21/org/apache/lucene/store/MemorySegmentIndexInputProvider.java: ## @@ -92,10 +110,22 @@ private final MemorySegment[] map( } i

Re: [PR] DRAFT: Add support for posix_madvise to Java 21 MMapDirectory [lucene]

2024-03-22 Thread via GitHub
uschindler commented on PR #13196: URL: https://github.com/apache/lucene/pull/13196#issuecomment-2014791894 I refactored the code a bit: - The POSIX constants are now part of PosixNativeAcess class - The abstract method madvise() now only takes a `MemorySegment` and the `IOContext`. So

Re: [PR] DRAFT: Add support for posix_madvise to Java 21 MMapDirectory [lucene]

2024-03-22 Thread via GitHub
jpountz commented on code in PR #13196: URL: https://github.com/apache/lucene/pull/13196#discussion_r1535351343 ## lucene/core/src/java/org/apache/lucene/store/IOContext.java: ## @@ -54,14 +60,16 @@ public enum Context { public static final IOContext DEFAULT = new IOContext

Re: [PR] DRAFT: Add support for posix_madvise to Java 21 MMapDirectory [lucene]

2024-03-22 Thread via GitHub
uschindler commented on code in PR #13196: URL: https://github.com/apache/lucene/pull/13196#discussion_r1535348590 ## lucene/core/src/java/org/apache/lucene/store/IOContext.java: ## @@ -54,14 +60,16 @@ public enum Context { public static final IOContext DEFAULT = new IOCont

Re: [PR] DRAFT: Add support for posix_madvise to Java 21 MMapDirectory [lucene]

2024-03-22 Thread via GitHub
uschindler commented on code in PR #13196: URL: https://github.com/apache/lucene/pull/13196#discussion_r1535344579 ## lucene/core/src/java21/org/apache/lucene/store/MemorySegmentIndexInputProvider.java: ## @@ -92,10 +110,22 @@ private final MemorySegment[] map( } i

Re: [PR] DRAFT: Add support for posix_madvise to Java 21 MMapDirectory [lucene]

2024-03-22 Thread via GitHub
jpountz commented on code in PR #13196: URL: https://github.com/apache/lucene/pull/13196#discussion_r1535342453 ## lucene/core/src/java/org/apache/lucene/store/IOContext.java: ## @@ -54,14 +60,16 @@ public enum Context { public static final IOContext DEFAULT = new IOContext

Re: [PR] DRAFT: Add support for posix_madvise to Java 21 MMapDirectory [lucene]

2024-03-22 Thread via GitHub
ChrisHegarty commented on code in PR #13196: URL: https://github.com/apache/lucene/pull/13196#discussion_r1535331259 ## lucene/core/src/java21/org/apache/lucene/store/MemorySegmentIndexInputProvider.java: ## @@ -92,10 +110,22 @@ private final MemorySegment[] map( }

Re: [PR] DRAFT: Add support for posix_madvise to Java 21 MMapDirectory [lucene]

2024-03-22 Thread via GitHub
jpountz commented on code in PR #13196: URL: https://github.com/apache/lucene/pull/13196#discussion_r1535326207 ## lucene/core/src/java21/org/apache/lucene/store/MemorySegmentIndexInputProvider.java: ## @@ -92,10 +110,22 @@ private final MemorySegment[] map( } if (

Re: [PR] DRAFT: Add support for posix_madvise to Java 21 MMapDirectory [lucene]

2024-03-22 Thread via GitHub
jpountz commented on code in PR #13196: URL: https://github.com/apache/lucene/pull/13196#discussion_r1535324025 ## lucene/core/src/java/org/apache/lucene/store/IOContext.java: ## @@ -54,14 +60,16 @@ public enum Context { public static final IOContext DEFAULT = new IOContext

Re: [PR] DRAFT: Add support for posix_madvise to Java 21 MMapDirectory [lucene]

2024-03-22 Thread via GitHub
ChrisHegarty commented on code in PR #13196: URL: https://github.com/apache/lucene/pull/13196#discussion_r1535309204 ## lucene/core/src/java/org/apache/lucene/store/IOContext.java: ## @@ -54,14 +60,16 @@ public enum Context { public static final IOContext DEFAULT = new IOCo

Re: [PR] DRAFT: Add support for posix_madvise to Java 21 MMapDirectory [lucene]

2024-03-22 Thread via GitHub
ChrisHegarty commented on code in PR #13196: URL: https://github.com/apache/lucene/pull/13196#discussion_r1535291132 ## lucene/core/src/java21/org/apache/lucene/store/MemorySegmentIndexInputProvider.java: ## @@ -92,10 +110,22 @@ private final MemorySegment[] map( }

Re: [PR] DRAFT: Add support for posix_madvise to Java 21 MMapDirectory [lucene]

2024-03-22 Thread via GitHub
ChrisHegarty commented on code in PR #13196: URL: https://github.com/apache/lucene/pull/13196#discussion_r1535283798 ## lucene/core/src/java21/org/apache/lucene/store/PosixNativeAccess.java: ## @@ -0,0 +1,82 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one o

Re: [PR] DRAFT: Add support for posix_madvise to Java 21 MMapDirectory [lucene]

2024-03-22 Thread via GitHub
uschindler commented on code in PR #13196: URL: https://github.com/apache/lucene/pull/13196#discussion_r1535283242 ## lucene/core/src/java21/org/apache/lucene/store/MemorySegmentIndexInputProvider.java: ## @@ -92,10 +110,22 @@ private final MemorySegment[] map( } i

Re: [PR] DRAFT: Add support for posix_madvise to Java 21 MMapDirectory [lucene]

2024-03-22 Thread via GitHub
uschindler commented on code in PR #13196: URL: https://github.com/apache/lucene/pull/13196#discussion_r1535276508 ## lucene/core/src/java21/org/apache/lucene/store/PosixNativeAccess.java: ## @@ -0,0 +1,82 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or

Re: [PR] DRAFT: Add support for posix_madvise to Java 21 MMapDirectory [lucene]

2024-03-22 Thread via GitHub
uschindler commented on code in PR #13196: URL: https://github.com/apache/lucene/pull/13196#discussion_r1535274772 ## lucene/core/src/java21/org/apache/lucene/store/PosixNativeAccess.java: ## @@ -0,0 +1,82 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or

[PR] Break point estimate when threshold exceeded [lucene]

2024-03-22 Thread via GitHub
gf2121 opened a new pull request, #13199: URL: https://github.com/apache/lucene/pull/13199 Typically, we estimate the point value count to compare to a threshold and all we need is just a boolean which represents whether the point count is greater than this threshold. This pr proposes to pa

Re: [PR] DRAFT: Add support for posix_madvise to Java 21 MMapDirectory [lucene]

2024-03-22 Thread via GitHub
uschindler commented on code in PR #13196: URL: https://github.com/apache/lucene/pull/13196#discussion_r1535248163 ## lucene/core/src/java21/org/apache/lucene/store/MemorySegmentIndexInputProvider.java: ## @@ -92,10 +110,22 @@ private final MemorySegment[] map( } i

Re: [PR] DRAFT: Add support for posix_madvise to Java 21 MMapDirectory [lucene]

2024-03-22 Thread via GitHub
jpountz commented on code in PR #13196: URL: https://github.com/apache/lucene/pull/13196#discussion_r1535211651 ## lucene/core/src/java/org/apache/lucene/store/IOContext.java: ## @@ -54,14 +60,16 @@ public enum Context { public static final IOContext DEFAULT = new IOContext

Re: [PR] DRAFT: Add support for posix_madvise to Java 21 MMapDirectory [lucene]

2024-03-22 Thread via GitHub
jpountz commented on code in PR #13196: URL: https://github.com/apache/lucene/pull/13196#discussion_r1535211651 ## lucene/core/src/java/org/apache/lucene/store/IOContext.java: ## @@ -54,14 +60,16 @@ public enum Context { public static final IOContext DEFAULT = new IOContext

[PR] Fix TestTaxonomyFacetValueSource.testRandom [lucene]

2024-03-22 Thread via GitHub
iamsanjay opened a new pull request, #13198: URL: https://github.com/apache/lucene/pull/13198 Fix #13191 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To