Re: [PR] prefetch may select the wrong memory segment for multi-segment slices [lucene]

2025-01-08 Thread via GitHub
ChrisHegarty merged PR #14109: URL: https://github.com/apache/lucene/pull/14109 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucen

Re: [I] [Discuss] Reducing allocations in HnswUtil::markRooted [lucene]

2025-01-08 Thread via GitHub
viswanathk closed issue #14002: [Discuss] Reducing allocations in HnswUtil::markRooted URL: https://github.com/apache/lucene/issues/14002 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [I] [Discuss] Reducing allocations in HnswUtil::markRooted [lucene]

2025-01-08 Thread via GitHub
viswanathk commented on issue #14002: URL: https://github.com/apache/lucene/issues/14002#issuecomment-2578000965 Allocations are negligible after #14022. Closing this thread. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

Re: [I] Unable to launch Luke in Windows 11 [lucene]

2025-01-08 Thread via GitHub
pzygielo commented on issue #14114: URL: https://github.com/apache/lucene/issues/14114#issuecomment-2578394124 Perhaps: lucene-core-10.1.0.jar: ``` org/apache/lucene/geo/Rectangle.class: compiled Java class data, version 65.0 ``` \+ ![image](https://github.com/u

Re: [I] Unable to launch Luke in Windows 11 [lucene]

2025-01-08 Thread via GitHub
pzygielo commented on issue #14114: URL: https://github.com/apache/lucene/issues/14114#issuecomment-2578406780 https://lucene.apache.org/core/10_1_0/SYSTEM_REQUIREMENTS.html: > Apache Lucene runs on Java 21 or greater. -- This is an automated message from the Apache Git Service. To

Re: [PR] Removing unnecessary ByteArrayDataInput allocations by resetting inplace [lucene]

2025-01-08 Thread via GitHub
iverase commented on code in PR #14113: URL: https://github.com/apache/lucene/pull/14113#discussion_r1907744833 ## lucene/core/src/java/org/apache/lucene/codecs/lucene90/Lucene90DocValuesProducer.java: ## @@ -1149,6 +1148,8 @@ private class TermsDict extends BaseTermsEnum {

Re: [PR] Implement ACORN-1 search for HNSW [lucene]

2025-01-08 Thread via GitHub
benwtrent commented on PR #14085: URL: https://github.com/apache/lucene/pull/14085#issuecomment-2578519157 Here are some benchmarks (100k float32[1024]). Baseline: ``` recall latency (ms)nDoc topK fanout maxConn beamWidth visited selectivity 0.915 0.950 1

Re: [I] Unable to launch Luke in Windows 11 [lucene]

2025-01-08 Thread via GitHub
dweiss commented on issue #14114: URL: https://github.com/apache/lucene/issues/14114#issuecomment-2578459925 Oh, I missed that in the description: ``` java version "1.8.0_431" ``` As @pzygielo pointed out, your Java is way, way outdated. Please download Java 21 or newer and t

Re: [I] Unable to launch Luke in Windows 11 [lucene]

2025-01-08 Thread via GitHub
dweiss closed issue #14114: Unable to launch Luke in Windows 11 URL: https://github.com/apache/lucene/issues/14114 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscrib

[PR] Use CDL to block threads to avoid flaky tests. [lucene]

2025-01-08 Thread via GitHub
aoli-al opened a new pull request, #14116: URL: https://github.com/apache/lucene/pull/14116 ### Description This PR fixes a flaky test in `testIntraMergeThreadPoolIsLimitedByMaxThreads`. The test uses sleep to block thread execution which is not deterministic due to concurrency issue

[PR] fix for gh#14110: stablize BpVectorReordered heuristic [lucene]

2025-01-08 Thread via GitHub
msokolov opened a new pull request, #14117: URL: https://github.com/apache/lucene/pull/14117 It turned out that the initial shuffled order resulted in a left / right split with centroids that were very close together, the heuristic would terminate without doing any swaps, causing the test t

[I] Unable to launch Luke in Windows 11 [lucene]

2025-01-08 Thread via GitHub
eqtr-ab opened a new issue, #14114: URL: https://github.com/apache/lucene/issues/14114 ### Description Issue: When attempting to run the Luke tool using "luke.cmd", the following error message is given: Error: Could not create the Java Virtual Machine. Error: A fatal exce

Re: [I] Unable to launch Luke in Windows 11 [lucene]

2025-01-08 Thread via GitHub
eqtr-ab commented on issue #14114: URL: https://github.com/apache/lucene/issues/14114#issuecomment-2577823580 Also worth noting that I've tried altering the script to use single dashes instead of double, for example: %LAUNCH_START% "%LAUNCH_CMD%" %LAUNCH_OPTS% -module-path "%MODULES%

Re: [PR] Add two new "Seeded" Knn queries for seeded vector search [lucene]

2025-01-08 Thread via GitHub
seanmacavaney commented on PR #14084: URL: https://github.com/apache/lucene/pull/14084#issuecomment-2577760107 Looks great, thanks @benwtrent! I'm keen to benchmark it a bit, but no need to hold up merging it over that. -- This is an automated message from the Apache Git Service. To respo

Re: [PR] Add some basic HNSW graph checks to CheckIndex [lucene]

2025-01-08 Thread via GitHub
mikemccand commented on PR #13984: URL: https://github.com/apache/lucene/pull/13984#issuecomment-2578347646 > Let me know if you're able to test this revision! Yay, I just tested again and I see HNSW output, and the nightly benchy index passed all the new tests here! Thanks @benchapl

Re: [I] Unable to launch Luke in Windows 11 [lucene]

2025-01-08 Thread via GitHub
dweiss commented on issue #14114: URL: https://github.com/apache/lucene/issues/14114#issuecomment-2578342845 Try opening cmd (command line interpreter) and run the following: ``` SET LAUNCH_CMD=javaw luke.cmd ``` What does it say? Is there any error? -- This is an automated

Re: [PR] Add some basic HNSW graph checks to CheckIndex [lucene]

2025-01-08 Thread via GitHub
msokolov merged PR #13984: URL: https://github.com/apache/lucene/pull/13984 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucene.ap

Re: [PR] Add some basic HNSW graph checks to CheckIndex [lucene]

2025-01-08 Thread via GitHub
msokolov commented on PR #13984: URL: https://github.com/apache/lucene/pull/13984#issuecomment-2578359028 FYI @benchaplin this is merged now, but then I realized there is no CHANGES entry -- would you like to add one? After that, we can probably also backport to `branch_10x` using cherry-pi

[I] backwards compat tests fail with "too many neighbors: 11" [lucene]

2025-01-08 Thread via GitHub
msokolov opened a new issue, #14118: URL: https://github.com/apache/lucene/issues/14118 ### Description repros for me with ``` gradlew :lucene:backward-codecs:test --tests "org.apache.lucene.backward_index.TestIndexUpgradeBackwardsCompatibility.testUpgradeOldIndex {Lucene-V

Re: [I] Unable to launch Luke in Windows 11 [lucene]

2025-01-08 Thread via GitHub
dweiss commented on issue #14114: URL: https://github.com/apache/lucene/issues/14114#issuecomment-2578372904 I've just tried on Windows 10, with Java 21 and everything seems to work fine. When you start cmd, does this work? ``` java -version ``` If it doesn't then perhaps there i

Re: [PR] Add two new "Seeded" Knn queries for seeded vector search [lucene]

2025-01-08 Thread via GitHub
benwtrent commented on code in PR #14084: URL: https://github.com/apache/lucene/pull/14084#discussion_r1907184892 ## lucene/core/src/java/org/apache/lucene/search/SeededKnnByteVectorQuery.java: ## @@ -0,0 +1,90 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under on

[I] occasional TestMonitorReadonly.testReadonlyMonitorGetsRefreshed failure [lucene]

2025-01-08 Thread via GitHub
cpoerschke opened a new issue, #14115: URL: https://github.com/apache/lucene/issues/14115 ### Description e.g. see https://lists.apache.org/list?bui...@lucene.apache.org:lte=1M:%22org.apache.lucene.monitor.TestMonitorReadonly.testReadonlyMonitorGetsRefreshed%22 ``` Stack Tra

Re: [I] occasional TestMonitorReadonly.testReadonlyMonitorGetsRefreshed failure [lucene]

2025-01-08 Thread via GitHub
cpoerschke commented on issue #14115: URL: https://github.com/apache/lucene/issues/14115#issuecomment-2577691288 Have not yet been able to reproduce this locally but from code reading wondering the an `onPurgeError` hook in the test code would be helpful alongside the `onPurge` hook here:

Re: [PR] Implement ACORN-1 search for HNSW [lucene]

2025-01-08 Thread via GitHub
benwtrent commented on PR #14085: URL: https://github.com/apache/lucene/pull/14085#issuecomment-2578229631 Hey @benchaplin there are a number of things broken with lucene util right now. Your recall numbers surprised me and I think they don't reflect actual performance. I am working

Re: [PR] Add two new "Seeded" Knn queries for seeded vector search [lucene]

2025-01-08 Thread via GitHub
benwtrent commented on code in PR #14084: URL: https://github.com/apache/lucene/pull/14084#discussion_r1907254811 ## lucene/core/src/java/org/apache/lucene/search/SeededKnnByteVectorQuery.java: ## @@ -0,0 +1,90 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under on

Re: [PR] Preserve max-conn when merging onto existing graph Fixes gh#14118 [lucene]

2025-01-08 Thread via GitHub
msokolov commented on PR #14121: URL: https://github.com/apache/lucene/pull/14121#issuecomment-2578623132 fixes #14118 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To u

Re: [I] backwards compat tests fail with "too many neighbors: 11" [lucene]

2025-01-08 Thread via GitHub
msokolov commented on issue #14118: URL: https://github.com/apache/lucene/issues/14118#issuecomment-2578618058 It seems this was a bug introduced with https://github.com/apache/lucene/pull/14097 where we made a subtle change to preserve HNSW graph max-connections parameter from existing gra

Re: [PR] Implement ACORN-1 search for HNSW [lucene]

2025-01-08 Thread via GitHub
benwtrent commented on PR #14085: URL: https://github.com/apache/lucene/pull/14085#issuecomment-2578508784 This is the branch I am using for testing recall/latency for filter cases for right now: https://github.com/mikemccand/luceneutil/compare/main...benwtrent:luceneutil:filter-testing?exp

[PR] Fix acceptOrds in EmptyOffHeapVectorValues to match no bits [lucene]

2025-01-08 Thread via GitHub
vigyasharma opened a new pull request, #14119: URL: https://github.com/apache/lucene/pull/14119 EmptyOffHeapVectorValues currently returns null for `getAcceptOrds()`. However, internally across the codebase, we assume that null `Bits` imply that all values are accepted – like null for `acce

[PR] Add CHANGES entry for CheckIndex HNSW work [lucene]

2025-01-08 Thread via GitHub
benchaplin opened a new pull request, #14120: URL: https://github.com/apache/lucene/pull/14120 ### Description https://github.com/apache/lucene/pull/13984 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

Re: [PR] Add some basic HNSW graph checks to CheckIndex [lucene]

2025-01-08 Thread via GitHub
benchaplin commented on PR #13984: URL: https://github.com/apache/lucene/pull/13984#issuecomment-2578611721 Thanks @mikemccand and @msokolov! And oh, forgot about the CHANGES - here's an entry (I listed under 10.2.0) https://github.com/apache/lucene/pull/14120. -- This is an automated mes

[PR] Preserve max-conn when merging onto existing graph Fixes gh#14118 [lucene]

2025-01-08 Thread via GitHub
msokolov opened a new pull request, #14121: URL: https://github.com/apache/lucene/pull/14121 preserve the HNSW graph max-conn when writing -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [PR] Implement ACORN-1 search for HNSW [lucene]

2025-01-08 Thread via GitHub
benwtrent commented on PR #14085: URL: https://github.com/apache/lucene/pull/14085#issuecomment-2578689141 https://github.com/apache/lucene/compare/main...benwtrent:lucene:acorn_search?expand=1 Here are two of my ideas: - We only go to 2-hop if a percentage of the current cand

Re: [PR] Add two new "Seeded" Knn queries for seeded vector search [lucene]

2025-01-08 Thread via GitHub
cpoerschke commented on code in PR #14084: URL: https://github.com/apache/lucene/pull/14084#discussion_r1906914231 ## lucene/core/src/java/org/apache/lucene/search/SeededKnnByteVectorQuery.java: ## @@ -0,0 +1,90 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under o

Re: [PR] Fix test that was implicitly assuming index would not be reordered [lucene]

2025-01-08 Thread via GitHub
msokolov merged PR #14122: URL: https://github.com/apache/lucene/pull/14122 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucene.ap

Re: [PR] Add CHANGES entry for CheckIndex HNSW work [lucene]

2025-01-08 Thread via GitHub
msokolov merged PR #14120: URL: https://github.com/apache/lucene/pull/14120 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucene.ap

[PR] Fix test that was implicitly assuming index would not be reordered [lucene]

2025-01-08 Thread via GitHub
msokolov opened a new pull request, #14122: URL: https://github.com/apache/lucene/pull/14122 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-

Re: [PR] fix for gh#14110: stablize BpVectorReordered heuristic [lucene]

2025-01-08 Thread via GitHub
msokolov commented on PR #14117: URL: https://github.com/apache/lucene/pull/14117#issuecomment-2578804815 BTW this also fixed failures like this one: ``` FAILED: org.apache.lucene.misc.index.TestBpVectorReorderer.testQuantizedIndex Error Message: java.lang.AssertionError