Re: [PR] Aggregate files from the same segment into a single Arena [lucene]

2024-07-15 Thread via GitHub
uschindler commented on code in PR #13570: URL: https://github.com/apache/lucene/pull/13570#discussion_r1677539871 ## lucene/core/src/java/org/apache/lucene/store/MMapDirectory.java: ## @@ -256,15 +267,15 @@ default IOException convertMapFailedIOException( } } - priva

Re: [PR] Aggregate files from the same segment into a single Arena [lucene]

2024-07-15 Thread via GitHub
uschindler commented on code in PR #13570: URL: https://github.com/apache/lucene/pull/13570#discussion_r1677544947 ## lucene/core/src/java/org/apache/lucene/store/MMapDirectory.java: ## @@ -199,20 +203,27 @@ public IndexInput openInput(String name, IOContext context) throws IOE

Re: [PR] Aggregate files from the same segment into a single Arena [lucene]

2024-07-15 Thread via GitHub
ChrisHegarty commented on code in PR #13570: URL: https://github.com/apache/lucene/pull/13570#discussion_r1677621308 ## lucene/core/src/java/org/apache/lucene/store/MMapDirectory.java: ## @@ -199,20 +203,27 @@ public IndexInput openInput(String name, IOContext context) throws I

Re: [PR] Aggregate files from the same segment into a single Arena [lucene]

2024-07-15 Thread via GitHub
ChrisHegarty commented on code in PR #13570: URL: https://github.com/apache/lucene/pull/13570#discussion_r1677620883 ## lucene/core/src/java/org/apache/lucene/store/MMapDirectory.java: ## @@ -199,20 +203,27 @@ public IndexInput openInput(String name, IOContext context) throws I

Re: [PR] Add levels to DocValues skipper index [lucene]

2024-07-15 Thread via GitHub
iverase commented on code in PR #13563: URL: https://github.com/apache/lucene/pull/13563#discussion_r1677624311 ## lucene/core/src/java/org/apache/lucene/codecs/lucene90/Lucene90DocValuesProducer.java: ## @@ -1792,61 +1794,88 @@ public DocValuesSkipper getSkipper(FieldInfo field

Re: [PR] Compute facets while collecting [lucene]

2024-07-15 Thread via GitHub
stefanvodita commented on code in PR #13568: URL: https://github.com/apache/lucene/pull/13568#discussion_r1677574763 ## lucene/facet/src/java/org/apache/lucene/facet/DrillSideways.java: ## @@ -300,35 +302,28 @@ public DrillSidewaysResult( } } - private static class Ca

Re: [PR] Add levels to DocValues skipper index [lucene]

2024-07-15 Thread via GitHub
jpountz commented on code in PR #13563: URL: https://github.com/apache/lucene/pull/13563#discussion_r1677636214 ## lucene/core/src/java/org/apache/lucene/codecs/lucene90/Lucene90DocValuesProducer.java: ## @@ -1792,61 +1794,88 @@ public DocValuesSkipper getSkipper(FieldInfo field

Re: [PR] Add levels to DocValues skipper index [lucene]

2024-07-15 Thread via GitHub
iverase commented on code in PR #13563: URL: https://github.com/apache/lucene/pull/13563#discussion_r1677637583 ## lucene/core/src/java/org/apache/lucene/codecs/lucene90/Lucene90DocValuesConsumer.java: ## @@ -207,65 +210,120 @@ void accumulate(long value) { maxValue = Mat

Re: [PR] Aggregate files from the same segment into a single Arena [lucene]

2024-07-15 Thread via GitHub
ChrisHegarty commented on code in PR #13570: URL: https://github.com/apache/lucene/pull/13570#discussion_r1677637602 ## lucene/core/src/java21/org/apache/lucene/store/RefCountedSharedArena.java: ## @@ -0,0 +1,112 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under

Re: [PR] Add levels to DocValues skipper index [lucene]

2024-07-15 Thread via GitHub
iverase commented on code in PR #13563: URL: https://github.com/apache/lucene/pull/13563#discussion_r1677661302 ## lucene/core/src/java/org/apache/lucene/codecs/lucene90/Lucene90DocValuesProducer.java: ## @@ -1792,61 +1794,88 @@ public DocValuesSkipper getSkipper(FieldInfo field

Re: [PR] Compute facets while collecting [lucene]

2024-07-15 Thread via GitHub
epotyom commented on PR #13568: URL: https://github.com/apache/lucene/pull/13568#issuecomment-2228412566 @stefanvodita thank you for reviewing! > Do you have any benchmark results to share? Hooking this into luceneutil might be tricky. I think even a vague sense of where this is bett

Re: [PR] Compute facets while collecting [lucene]

2024-07-15 Thread via GitHub
epotyom commented on code in PR #13568: URL: https://github.com/apache/lucene/pull/13568#discussion_r166951 ## lucene/core/src/java/org/apache/lucene/search/CollectorOwner.java: ## @@ -0,0 +1,86 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more +

Re: [PR] Compute facets while collecting [lucene]

2024-07-15 Thread via GitHub
epotyom commented on code in PR #13568: URL: https://github.com/apache/lucene/pull/13568#discussion_r1677782365 ## lucene/core/src/java/org/apache/lucene/search/CollectorOwner.java: ## @@ -0,0 +1,86 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more +

Re: [PR] SparseFixedBitSet#firstDoc: reduce number of `indices` iterations for a bit set that is not fully built yet. [lucene]

2024-07-15 Thread via GitHub
msokolov commented on PR #13559: URL: https://github.com/apache/lucene/pull/13559#issuecomment-2228466558 Another idea -- would it help your use case? -- would be to support `nextSetBit(start, end)` . We could do this without adding any additional tracking in existing SparseBitSet methods

Re: [PR] SparseFixedBitSet#firstDoc: reduce number of `indices` iterations for a bit set that is not fully built yet. [lucene]

2024-07-15 Thread via GitHub
msokolov commented on code in PR #13559: URL: https://github.com/apache/lucene/pull/13559#discussion_r1677810270 ## lucene/core/src/java/org/apache/lucene/util/SparseFixedBitSet.java: ## @@ -51,6 +51,7 @@ private static int blockCount(int length) { } final long[] indices

Re: [PR] SparseFixedBitSet#firstDoc: reduce number of `indices` iterations for a bit set that is not fully built yet. [lucene]

2024-07-15 Thread via GitHub
epotyom commented on PR #13559: URL: https://github.com/apache/lucene/pull/13559#issuecomment-2228524276 @msokolov thanks for looking into it! > I wonder if DocIdSetBuilder would help? I guess it doesn't support accessing as a BitSet while under construction though. Yes, I thin

Re: [PR] SparseFixedBitSet#firstDoc: reduce number of `indices` iterations for a bit set that is not fully built yet. [lucene]

2024-07-15 Thread via GitHub
epotyom commented on code in PR #13559: URL: https://github.com/apache/lucene/pull/13559#discussion_r1677851990 ## lucene/core/src/java/org/apache/lucene/util/SparseFixedBitSet.java: ## @@ -51,6 +51,7 @@ private static int blockCount(int length) { } final long[] indices;

Re: [PR] SparseFixedBitSet#firstDoc: reduce number of `indices` iterations for a bit set that is not fully built yet. [lucene]

2024-07-15 Thread via GitHub
epotyom commented on code in PR #13559: URL: https://github.com/apache/lucene/pull/13559#discussion_r1677851990 ## lucene/core/src/java/org/apache/lucene/util/SparseFixedBitSet.java: ## @@ -51,6 +51,7 @@ private static int blockCount(int length) { } final long[] indices;

Re: [PR] SparseFixedBitSet#firstDoc: reduce number of `indices` iterations for a bit set that is not fully built yet. [lucene]

2024-07-15 Thread via GitHub
epotyom commented on code in PR #13559: URL: https://github.com/apache/lucene/pull/13559#discussion_r1677851990 ## lucene/core/src/java/org/apache/lucene/util/SparseFixedBitSet.java: ## @@ -51,6 +51,7 @@ private static int blockCount(int length) { } final long[] indices;

Re: [PR] Compute facets while collecting [lucene]

2024-07-15 Thread via GitHub
epotyom commented on code in PR #13568: URL: https://github.com/apache/lucene/pull/13568#discussion_r1677862194 ## lucene/core/src/java/org/apache/lucene/search/CollectorOwner.java: ## @@ -0,0 +1,86 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more +

Re: [PR] Compute facets while collecting [lucene]

2024-07-15 Thread via GitHub
epotyom commented on code in PR #13568: URL: https://github.com/apache/lucene/pull/13568#discussion_r1677870287 ## lucene/core/src/java/org/apache/lucene/search/DoubleValuesSource.java: ## @@ -115,6 +116,69 @@ public final LongValuesSource toLongValuesSource() { return new

Re: [PR] Compute facets while collecting [lucene]

2024-07-15 Thread via GitHub
epotyom commented on code in PR #13568: URL: https://github.com/apache/lucene/pull/13568#discussion_r1677872262 ## lucene/core/src/java/org/apache/lucene/search/IndexSearcher.java: ## @@ -630,27 +630,47 @@ private TopFieldDocs searchAfter( */ public T search(Query query,

Re: [PR] SparseFixedBitSet#firstDoc: reduce number of `indices` iterations for a bit set that is not fully built yet. [lucene]

2024-07-15 Thread via GitHub
msokolov commented on code in PR #13559: URL: https://github.com/apache/lucene/pull/13559#discussion_r1677885744 ## lucene/core/src/java/org/apache/lucene/util/SparseFixedBitSet.java: ## @@ -51,6 +51,7 @@ private static int blockCount(int length) { } final long[] indices

Re: [PR] Compute facets while collecting [lucene]

2024-07-15 Thread via GitHub
epotyom commented on code in PR #13568: URL: https://github.com/apache/lucene/pull/13568#discussion_r1677923025 ## lucene/sandbox/src/java/org/apache/lucene/sandbox/facet/ordinal_iterators/SortOrdinalIterator.java: ## @@ -0,0 +1,85 @@ +/* + * Licensed to the Apache Software Foun

Re: [PR] Compute facets while collecting [lucene]

2024-07-15 Thread via GitHub
epotyom commented on code in PR #13568: URL: https://github.com/apache/lucene/pull/13568#discussion_r1677925857 ## lucene/sandbox/src/java/org/apache/lucene/sandbox/facet/ordinal_iterators/SortOrdinalIterator.java: ## @@ -0,0 +1,85 @@ +/* + * Licensed to the Apache Software Foun

Re: [PR] Aggregate files from the same segment into a single Arena [lucene]

2024-07-15 Thread via GitHub
magibney commented on code in PR #13570: URL: https://github.com/apache/lucene/pull/13570#discussion_r1677916943 ## lucene/core/src/java21/org/apache/lucene/store/RefCountedSharedArena.java: ## @@ -0,0 +1,102 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one

Re: [PR] Aggregate files from the same segment into a single Arena [lucene]

2024-07-15 Thread via GitHub
uschindler commented on code in PR #13570: URL: https://github.com/apache/lucene/pull/13570#discussion_r1677946847 ## lucene/core/src/java21/org/apache/lucene/store/RefCountedSharedArena.java: ## @@ -0,0 +1,102 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under on

Re: [PR] Aggregate files from the same segment into a single Arena [lucene]

2024-07-15 Thread via GitHub
uschindler commented on code in PR #13570: URL: https://github.com/apache/lucene/pull/13570#discussion_r1677959677 ## lucene/core/src/java21/org/apache/lucene/store/RefCountedSharedArena.java: ## @@ -0,0 +1,102 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under on

Re: [PR] Aggregate files from the same segment into a single Arena [lucene]

2024-07-15 Thread via GitHub
uschindler commented on code in PR #13570: URL: https://github.com/apache/lucene/pull/13570#discussion_r1677959677 ## lucene/core/src/java21/org/apache/lucene/store/RefCountedSharedArena.java: ## @@ -0,0 +1,102 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under on

Re: [PR] Add IntervalsSource for range and regexp queries (#13562) [lucene]

2024-07-15 Thread via GitHub
mayya-sharipova merged PR #13569: URL: https://github.com/apache/lucene/pull/13569 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lu

Re: [PR] Aggregate files from the same segment into a single Arena [lucene]

2024-07-15 Thread via GitHub
uschindler commented on PR #13570: URL: https://github.com/apache/lucene/pull/13570#issuecomment-2228727994 Hi, we also get Updates on the JDK fixes: https://github.com/openjdk/jdk/pull/20158 (see issue about more details and some improvements Solr and/or Elasticsearch/Opensearch should

Re: [PR] Aggregate files from the same segment into a single Arena [lucene]

2024-07-15 Thread via GitHub
ChrisHegarty commented on code in PR #13570: URL: https://github.com/apache/lucene/pull/13570#discussion_r1677994244 ## lucene/core/src/java21/org/apache/lucene/store/RefCountedSharedArena.java: ## @@ -0,0 +1,102 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under

Re: [I] Examine performance of individual data accessor methods of MemorySegmentIndexInput when IndexInputs are closed in other threads (deoptimizations,...) [lucene]

2024-07-15 Thread via GitHub
uschindler commented on issue #13325: URL: https://github.com/apache/lucene/issues/13325#issuecomment-2228749316 We also have another PR doing the same than @magibney's: https://github.com/apache/lucene/pull/13570 We also got some updates on the JDK front. Next to the fixes here (use

Re: [PR] Aggregate files from the same segment into a single Arena [lucene]

2024-07-15 Thread via GitHub
magibney commented on PR #13570: URL: https://github.com/apache/lucene/pull/13570#issuecomment-2228755150 To expand a bit on the concern I raised above: IIUC, in order for this to work properly (guaranteed to not potentially leak virtual memory address space) it depends on segment fil

Re: [PR] Aggregate files from the same segment into a single Arena [lucene]

2024-07-15 Thread via GitHub
uschindler commented on PR #13570: URL: https://github.com/apache/lucene/pull/13570#issuecomment-2228783249 > To expand a bit on the concern I raised above: > > IIUC, in order for this to work properly (guaranteed to not potentially leak virtual memory address space) it depends on seg

Re: [PR] Aggregate files from the same segment into a single Arena [lucene]

2024-07-15 Thread via GitHub
ChrisHegarty commented on code in PR #13570: URL: https://github.com/apache/lucene/pull/13570#discussion_r1678024394 ## lucene/core/src/java21/org/apache/lucene/store/RefCountedSharedArena.java: ## @@ -0,0 +1,102 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under

Re: [PR] Aggregate files from the same segment into a single Arena [lucene]

2024-07-15 Thread via GitHub
uschindler commented on PR #13570: URL: https://github.com/apache/lucene/pull/13570#issuecomment-2228809498 We may need some other logic: We have some files that can change without the segment name changing? How do we handle *.del od docValues update files? If they correspond to the same se

Re: [PR] Aggregate files from the same segment into a single Arena [lucene]

2024-07-15 Thread via GitHub
magibney commented on PR #13570: URL: https://github.com/apache/lucene/pull/13570#issuecomment-2228833300 heh, was just writing to suggest something similar (`Function>`). SGTM. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

Re: [PR] Group memory arenas by segment to reduce costly `Arena.close()` [lucene]

2024-07-15 Thread via GitHub
magibney closed pull request #13555: Group memory arenas by segment to reduce costly `Arena.close()` URL: https://github.com/apache/lucene/pull/13555 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

Re: [PR] Group memory arenas by segment to reduce costly `Arena.close()` [lucene]

2024-07-15 Thread via GitHub
magibney commented on PR #13555: URL: https://github.com/apache/lucene/pull/13555#issuecomment-2228844256 Closing in favor of #13570 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [I] Examine performance of individual data accessor methods of MemorySegmentIndexInput when IndexInputs are closed in other threads (deoptimizations,...) [lucene]

2024-07-15 Thread via GitHub
uschindler commented on issue #13325: URL: https://github.com/apache/lucene/issues/13325#issuecomment-2228936957 Yeah: https://github.com/openjdk/jdk/pull/20158#issuecomment-2228916752 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

Re: [I] Significant drop in recall for 8 bit Scalar Quantizer [lucene]

2024-07-15 Thread via GitHub
benwtrent commented on issue #13519: URL: https://github.com/apache/lucene/issues/13519#issuecomment-2228966111 @MilindShyani OK, I did some more benchmarking. I tried switching to `+` & with your full correction term and recall significantly dropped to `0.518` I tried the fol

Re: [PR] Introduces efSearch as a separate parameter in KNN{Byte:Float}VectorQuery [lucene]

2024-07-15 Thread via GitHub
shatejas commented on PR #13407: URL: https://github.com/apache/lucene/pull/13407#issuecomment-2228981677 @benwtrent > Its not necessary and without it things are simpler (less code and less API features is a good thing by itself) This shouldn't come at the cost of not providi

Re: [PR] Introduces efSearch as a separate parameter in KNN{Byte:Float}VectorQuery [lucene]

2024-07-15 Thread via GitHub
shatejas closed pull request #13407: Introduces efSearch as a separate parameter in KNN{Byte:Float}VectorQuery URL: https://github.com/apache/lucene/pull/13407 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

Re: [PR] Compute facets while collecting [lucene]

2024-07-15 Thread via GitHub
epotyom commented on code in PR #13568: URL: https://github.com/apache/lucene/pull/13568#discussion_r1678175501 ## lucene/sandbox/src/test/org/apache/lucene/sandbox/facet/SandboxFacetTestCase.java: ## @@ -0,0 +1,407 @@ +/* + * Licensed to the Apache Software Foundation (ASF) und

Re: [PR] Compute facets while collecting [lucene]

2024-07-15 Thread via GitHub
epotyom commented on code in PR #13568: URL: https://github.com/apache/lucene/pull/13568#discussion_r1678177857 ## lucene/facet/src/java/org/apache/lucene/facet/DrillSideways.java: ## @@ -300,35 +302,28 @@ public DrillSidewaysResult( } } - private static class Callabl

Re: [PR] Compute facets while collecting [lucene]

2024-07-15 Thread via GitHub
epotyom commented on code in PR #13568: URL: https://github.com/apache/lucene/pull/13568#discussion_r1678181693 ## lucene/facet/src/java/org/apache/lucene/facet/DrillSideways.java: ## @@ -349,45 +344,149 @@ private DrillDownQuery getDrillDownQuery( public ConcurrentDrillSide

Re: [PR] Compute facets while collecting [lucene]

2024-07-15 Thread via GitHub
epotyom commented on code in PR #13568: URL: https://github.com/apache/lucene/pull/13568#discussion_r1678184630 ## lucene/facet/src/java/org/apache/lucene/facet/DrillSideways.java: ## @@ -398,130 +497,56 @@ private ConcurrentDrillSidewaysResult searchSequentially( }

Re: [PR] Compute facets while collecting [lucene]

2024-07-15 Thread via GitHub
epotyom commented on code in PR #13568: URL: https://github.com/apache/lucene/pull/13568#discussion_r1678185585 ## lucene/facet/src/java/org/apache/lucene/facet/DrillSideways.java: ## @@ -398,130 +497,56 @@ private ConcurrentDrillSidewaysResult searchSequentially( }

Re: [PR] Compute facets while collecting [lucene]

2024-07-15 Thread via GitHub
epotyom commented on code in PR #13568: URL: https://github.com/apache/lucene/pull/13568#discussion_r1678226435 ## lucene/facet/src/java/org/apache/lucene/facet/DrillSidewaysQuery.java: ## @@ -195,11 +156,11 @@ public ScorerSupplier scorerSupplier(LeafReaderContext context) thr

Re: [PR] Compute facets while collecting [lucene]

2024-07-15 Thread via GitHub
epotyom commented on code in PR #13568: URL: https://github.com/apache/lucene/pull/13568#discussion_r1678240834 ## lucene/sandbox/src/test/org/apache/lucene/sandbox/facet/TestRangeFacet.java: ## @@ -0,0 +1,1654 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under on

Re: [PR] Compute facets while collecting [lucene]

2024-07-15 Thread via GitHub
epotyom commented on code in PR #13568: URL: https://github.com/apache/lucene/pull/13568#discussion_r1678246773 ## lucene/sandbox/src/test/org/apache/lucene/sandbox/facet/TestRangeFacet.java: ## @@ -0,0 +1,1654 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under on

Re: [PR] Compute facets while collecting [lucene]

2024-07-15 Thread via GitHub
epotyom commented on code in PR #13568: URL: https://github.com/apache/lucene/pull/13568#discussion_r1678247836 ## lucene/sandbox/src/test/org/apache/lucene/sandbox/facet/TestRangeFacet.java: ## @@ -0,0 +1,1654 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under on

Re: [PR] Compute facets while collecting [lucene]

2024-07-15 Thread via GitHub
epotyom commented on code in PR #13568: URL: https://github.com/apache/lucene/pull/13568#discussion_r1678265251 ## lucene/sandbox/src/test/org/apache/lucene/sandbox/facet/TestTaxonomyFacet.java: ## @@ -0,0 +1,163 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under

Re: [PR] Aggregate files from the same segment into a single Arena [lucene]

2024-07-15 Thread via GitHub
dsmiley commented on code in PR #13570: URL: https://github.com/apache/lucene/pull/13570#discussion_r1678448829 ## lucene/core/src/java/org/apache/lucene/store/MMapDirectory.java: ## @@ -199,20 +202,27 @@ public IndexInput openInput(String name, IOContext context) throws IOExce

Re: [PR] Aggregate files from the same segment into a single Arena [lucene]

2024-07-15 Thread via GitHub
dsmiley commented on code in PR #13570: URL: https://github.com/apache/lucene/pull/13570#discussion_r1678459190 ## lucene/core/src/java21/org/apache/lucene/store/MemorySegmentIndexInputProvider.java: ## @@ -125,4 +134,31 @@ private final MemorySegment[] map( } return s

Re: [I] Examine performance of individual data accessor methods of MemorySegmentIndexInput when IndexInputs are closed in other threads (deoptimizations,...) [lucene]

2024-07-15 Thread via GitHub
dsmiley commented on issue #13325: URL: https://github.com/apache/lucene/issues/13325#issuecomment-2229550743 What does "refreshing indexes" mean exactly? Solr does not have special threads for opening/closing indexes; they will typically happen on the common Jetty request thread pool. If

Re: [PR] Add a clear button for log content [lucene]

2024-07-15 Thread via GitHub
github-actions[bot] commented on PR #13516: URL: https://github.com/apache/lucene/pull/13516#issuecomment-2229712938 This PR has not had activity in the past 2 weeks, labeling it as stale. If the PR is waiting for review, notify the d...@lucene.apache.org list. Thank you for your contributi

Re: [PR] Simplifying text area stream in Luke- ticket 12809 [lucene]

2024-07-15 Thread via GitHub
github-actions[bot] commented on PR #12869: URL: https://github.com/apache/lucene/pull/12869#issuecomment-2229714723 This PR has not had activity in the past 2 weeks, labeling it as stale. If the PR is waiting for review, notify the d...@lucene.apache.org list. Thank you for your contributi

[PR] New JMH benchmark method - vdot8s that implement int8 dotProduct in C… [lucene]

2024-07-15 Thread via GitHub
goankur opened a new pull request, #13572: URL: https://github.com/apache/lucene/pull/13572 Credit: https://www.elastic.co/search-labs/blog/vector-similarity-computations-ludicrous-speed Implement vectorized dot product in native C code using Neon intrinsics ### Descri

Re: [I] Significant drop in recall for 8 bit Scalar Quantizer [lucene]

2024-07-15 Thread via GitHub
MilindShyani commented on issue #13519: URL: https://github.com/apache/lucene/issues/13519#issuecomment-2229864587 Hi @benwtrent, I am trying to carefully check the math vs code. The first thing I noticed is that my definitions of dxs and dxq don't necessarily match with yours. That might b

Re: [PR] New JMH benchmark method - vdot8s that implement int8 dotProduct in C… [lucene]

2024-07-15 Thread via GitHub
rmuir commented on PR #13572: URL: https://github.com/apache/lucene/pull/13572#issuecomment-2230045004 Do we even need to use intrinsics? function is so simple that the compiler seems to do the right thing, e.g. use `SDOT` dot production instruction, given the correct flags: https://

Re: [PR] New JMH benchmark method - vdot8s that implement int8 dotProduct in C… [lucene]

2024-07-15 Thread via GitHub
rmuir commented on PR #13572: URL: https://github.com/apache/lucene/pull/13572#issuecomment-2230061569 I haven't benchmarked, just seems `SDOT` is the one to optimize for, and GCC can both recognize the code shape and autovectorize to it without hassle. my cheap 2021 phone has `asimd

Re: [PR] New JMH benchmark method - vdot8s that implement int8 dotProduct in C… [lucene]

2024-07-15 Thread via GitHub
rmuir commented on code in PR #13572: URL: https://github.com/apache/lucene/pull/13572#discussion_r1678790088 ## lucene/core/build.gradle: ## @@ -14,10 +14,43 @@ * See the License for the specific language governing permissions and * limitations under the License. */ +plug

Re: [PR] New JMH benchmark method - vdot8s that implement int8 dotProduct in C… [lucene]

2024-07-15 Thread via GitHub
rmuir commented on code in PR #13572: URL: https://github.com/apache/lucene/pull/13572#discussion_r1678799350 ## lucene/core/build.gradle: ## @@ -14,10 +14,43 @@ * See the License for the specific language governing permissions and * limitations under the License. */ +plug

Re: [PR] Aggregate files from the same segment into a single Arena [lucene]

2024-07-15 Thread via GitHub
uschindler commented on code in PR #13570: URL: https://github.com/apache/lucene/pull/13570#discussion_r1678835899 ## lucene/core/src/java21/org/apache/lucene/store/MemorySegmentIndexInputProvider.java: ## @@ -125,4 +134,31 @@ private final MemorySegment[] map( } retur

Re: [PR] Aggregate files from the same segment into a single Arena [lucene]

2024-07-15 Thread via GitHub
uschindler commented on code in PR #13570: URL: https://github.com/apache/lucene/pull/13570#discussion_r1678835899 ## lucene/core/src/java21/org/apache/lucene/store/MemorySegmentIndexInputProvider.java: ## @@ -125,4 +134,31 @@ private final MemorySegment[] map( } retur

Re: [I] Examine performance of individual data accessor methods of MemorySegmentIndexInput when IndexInputs are closed in other threads (deoptimizations,...) [lucene]

2024-07-15 Thread via GitHub
uschindler commented on issue #13325: URL: https://github.com/apache/lucene/issues/13325#issuecomment-2230135534 It is not a JDK bug, it is expected behaviour. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL a

Re: [I] Examine performance of individual data accessor methods of MemorySegmentIndexInput when IndexInputs are closed in other threads (deoptimizations,...) [lucene]

2024-07-15 Thread via GitHub
uschindler commented on issue #13325: URL: https://github.com/apache/lucene/issues/13325#issuecomment-2230139147 Basically the idea is to close indexes asynchronous in a single thread doing this to not block main threads. -- This is an automated message from the Apache Git Service. To res