uschindler commented on code in PR #13570:
URL: https://github.com/apache/lucene/pull/13570#discussion_r1687970721
##########
lucene/core/src/java/org/apache/lucene/store/MMapDirectory.java:
##########
@@ -83,6 +93,26 @@ public class MMapDirectory extends FSDirectory {
*/
public static final BiPredicate<String, IOContext> NO_FILES = (filename,
context) -> false;
+ /** Argument for {@link #setGroupingFunction(Function)} that configures no
grouping. */
+ public static final Function<String, Optional<String>> NO_GROUPING =
filename -> Optional.empty();
+
+ /** Argument for {@link #setGroupingFunction(Function)} that configures
grouping by segment. */
+ public static final Function<String, Optional<String>> GROUP_BY_SEGMENT =
+ filename -> {
+ if (!CODEC_FILE_PATTERN.matcher(filename).matches()) {
+ return Optional.empty();
+ }
+ String groupKey =
IndexFileNames.parseSegmentName(filename).substring(1);
+ try {
Review Comment:
Now that we have a maximum number of increments per segment, we could easily
handle all generations!=0 as one shared arena.
So when new deletions are added and the segment reloaded, it would create a
new arena for all changes to segment up to the total number of updates.
##########
lucene/core/src/java21/org/apache/lucene/store/MemorySegmentIndexInputProvider.java:
##########
@@ -125,4 +135,77 @@ private final MemorySegment[] map(
}
return segments;
}
+
+ @Override
+ public ConcurrentHashMap<String, RefCountedSharedArena> attachment() {
+ return new ConcurrentHashMap<>();
+ }
+
+ /**
+ * This sysprop allows to control the max number of permits that a
RefCountedSharedArena will
+ * support for its lifetime. For example, to set the max number of permits
to 256, pass the
+ * following on the command line pass {@code
+ *
-Dorg.apache.lucene.store.MemorySegmentIndexInputProvider.sharedArenaMaxPermits=256}.
Review Comment:
I'd remove the internal class name and use MMapDirectory (see the example
from 9.x:
https://github.com/apache/lucene/blob/c23f6c09f7de569d48803712da1540d007746be6/lucene/core/src/java/org/apache/lucene/store/MMapDirectory.java#L159-L160)
and the code reading it.
##########
lucene/core/src/java/org/apache/lucene/store/MMapDirectory.java:
##########
@@ -83,6 +93,26 @@ public class MMapDirectory extends FSDirectory {
*/
public static final BiPredicate<String, IOContext> NO_FILES = (filename,
context) -> false;
+ /** Argument for {@link #setGroupingFunction(Function)} that configures no
grouping. */
+ public static final Function<String, Optional<String>> NO_GROUPING =
filename -> Optional.empty();
+
+ /** Argument for {@link #setGroupingFunction(Function)} that configures
grouping by segment. */
+ public static final Function<String, Optional<String>> GROUP_BY_SEGMENT =
+ filename -> {
+ if (!CODEC_FILE_PATTERN.matcher(filename).matches()) {
+ return Optional.empty();
+ }
+ String groupKey =
IndexFileNames.parseSegmentName(filename).substring(1);
+ try {
Review Comment:
So I think we should simply use two slots: "generation=0/invalid" vs.
"generation>0"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
