Re: [PR] Aggregate files from the same segment into a single Arena [lucene]

2024-08-23 Thread via GitHub
uschindler commented on PR #13570: URL: https://github.com/apache/lucene/pull/13570#issuecomment-2306601362 We may need to change the changes entry to tell that to enduser. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and u

Re: [PR] Aggregate files from the same segment into a single Arena [lucene]

2024-08-22 Thread via GitHub
ChrisHegarty commented on PR #13570: URL: https://github.com/apache/lucene/pull/13570#issuecomment-2305406563 > @ChrisHegarty the PR ist listed in the 9.x section of changes. I could work on a backport (possibly tomorrow). I just want to make sure you haven't started. > > I am not ye

Re: [PR] Aggregate files from the same segment into a single Arena [lucene]

2024-08-22 Thread via GitHub
uschindler commented on PR #13570: URL: https://github.com/apache/lucene/pull/13570#issuecomment-2305145865 @ChrisHegarty the PR ist listed in the 9.x section of changes. I could work on a backport (possibly tomorrow). I just want to make sure you haven't started. I am not yet sure ho

Re: [PR] Aggregate files from the same segment into a single Arena [lucene]

2024-08-01 Thread via GitHub
uschindler commented on PR #13570: URL: https://github.com/apache/lucene/pull/13570#issuecomment-2264150313 Do we have a backport PR? Should I work on it? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above t

Re: [PR] Aggregate files from the same segment into a single Arena [lucene]

2024-07-25 Thread via GitHub
uschindler commented on PR #13570: URL: https://github.com/apache/lucene/pull/13570#issuecomment-2250030200 It's finally done. Thanks to all and help while implementing my idea! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

Re: [PR] Aggregate files from the same segment into a single Arena [lucene]

2024-07-25 Thread via GitHub
ChrisHegarty merged PR #13570: URL: https://github.com/apache/lucene/pull/13570 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucen

Re: [PR] Aggregate files from the same segment into a single Arena [lucene]

2024-07-25 Thread via GitHub
ChrisHegarty commented on code in PR #13570: URL: https://github.com/apache/lucene/pull/13570#discussion_r1691039509 ## lucene/core/src/java/org/apache/lucene/store/MMapDirectory.java: ## @@ -83,6 +94,41 @@ public class MMapDirectory extends FSDirectory { */ public static

Re: [PR] Aggregate files from the same segment into a single Arena [lucene]

2024-07-24 Thread via GitHub
dsmiley commented on code in PR #13570: URL: https://github.com/apache/lucene/pull/13570#discussion_r1690396863 ## lucene/core/src/java/org/apache/lucene/store/MMapDirectory.java: ## @@ -83,6 +94,41 @@ public class MMapDirectory extends FSDirectory { */ public static fina

Re: [PR] Aggregate files from the same segment into a single Arena [lucene]

2024-07-24 Thread via GitHub
uschindler commented on code in PR #13570: URL: https://github.com/apache/lucene/pull/13570#discussion_r1690264711 ## lucene/core/src/java/org/apache/lucene/store/MMapDirectory.java: ## @@ -83,6 +94,41 @@ public class MMapDirectory extends FSDirectory { */ public static f

Re: [PR] Aggregate files from the same segment into a single Arena [lucene]

2024-07-24 Thread via GitHub
dsmiley commented on code in PR #13570: URL: https://github.com/apache/lucene/pull/13570#discussion_r1690256783 ## lucene/core/src/java/org/apache/lucene/store/MMapDirectory.java: ## @@ -83,6 +94,41 @@ public class MMapDirectory extends FSDirectory { */ public static fina

Re: [PR] Aggregate files from the same segment into a single Arena [lucene]

2024-07-24 Thread via GitHub
uschindler commented on PR #13570: URL: https://github.com/apache/lucene/pull/13570#issuecomment-2248449684 > Otherwise, I plan to merge tomorrow. And then figure out how to backport! Code duplication with Arena vs. Session hell! -- This is an automated message from the Apache Git S

Re: [PR] Aggregate files from the same segment into a single Arena [lucene]

2024-07-24 Thread via GitHub
ChrisHegarty commented on code in PR #13570: URL: https://github.com/apache/lucene/pull/13570#discussion_r1689827722 ## lucene/core/src/java21/org/apache/lucene/store/RefCountedSharedArena.java: ## @@ -0,0 +1,146 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under

Re: [PR] Aggregate files from the same segment into a single Arena [lucene]

2024-07-24 Thread via GitHub
magibney commented on code in PR #13570: URL: https://github.com/apache/lucene/pull/13570#discussion_r1689819985 ## lucene/core/src/java21/org/apache/lucene/store/RefCountedSharedArena.java: ## @@ -0,0 +1,146 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one

Re: [PR] Aggregate files from the same segment into a single Arena [lucene]

2024-07-24 Thread via GitHub
ChrisHegarty commented on PR #13570: URL: https://github.com/apache/lucene/pull/13570#issuecomment-2247914051 This looks like it's in good shape. @magibney Any final comments? Otherwise, I plan to merge tomorrow. And then figure out how to backport! -- This is an automated message from t

Re: [PR] Aggregate files from the same segment into a single Arena [lucene]

2024-07-24 Thread via GitHub
uschindler commented on code in PR #13570: URL: https://github.com/apache/lucene/pull/13570#discussion_r1689595022 ## lucene/core/src/java/org/apache/lucene/store/MMapDirectory.java: ## @@ -83,6 +94,38 @@ public class MMapDirectory extends FSDirectory { */ public static f

Re: [PR] Aggregate files from the same segment into a single Arena [lucene]

2024-07-24 Thread via GitHub
ChrisHegarty commented on code in PR #13570: URL: https://github.com/apache/lucene/pull/13570#discussion_r1689529488 ## lucene/core/src/java21/org/apache/lucene/store/MemorySegmentIndexInputProvider.java: ## @@ -125,4 +135,77 @@ private final MemorySegment[] map( } ret

Re: [PR] Aggregate files from the same segment into a single Arena [lucene]

2024-07-24 Thread via GitHub
ChrisHegarty commented on code in PR #13570: URL: https://github.com/apache/lucene/pull/13570#discussion_r1689527370 ## lucene/core/src/java/org/apache/lucene/store/MMapDirectory.java: ## @@ -83,6 +93,26 @@ public class MMapDirectory extends FSDirectory { */ public static

Re: [PR] Aggregate files from the same segment into a single Arena [lucene]

2024-07-23 Thread via GitHub
uschindler commented on code in PR #13570: URL: https://github.com/apache/lucene/pull/13570#discussion_r1688093358 ## lucene/core/src/java21/org/apache/lucene/store/MemorySegmentIndexInputProvider.java: ## @@ -125,4 +135,77 @@ private final MemorySegment[] map( } retur

Re: [PR] Aggregate files from the same segment into a single Arena [lucene]

2024-07-23 Thread via GitHub
uschindler commented on code in PR #13570: URL: https://github.com/apache/lucene/pull/13570#discussion_r1687970721 ## lucene/core/src/java/org/apache/lucene/store/MMapDirectory.java: ## @@ -83,6 +93,26 @@ public class MMapDirectory extends FSDirectory { */ public static f

Re: [PR] Aggregate files from the same segment into a single Arena [lucene]

2024-07-23 Thread via GitHub
ChrisHegarty commented on code in PR #13570: URL: https://github.com/apache/lucene/pull/13570#discussion_r1687920309 ## lucene/core/src/java21/org/apache/lucene/store/RefCountedSharedArena.java: ## @@ -0,0 +1,138 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under

Re: [PR] Aggregate files from the same segment into a single Arena [lucene]

2024-07-23 Thread via GitHub
ChrisHegarty commented on code in PR #13570: URL: https://github.com/apache/lucene/pull/13570#discussion_r1687726079 ## lucene/core/src/java21/org/apache/lucene/store/RefCountedSharedArena.java: ## @@ -0,0 +1,138 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under

Re: [PR] Aggregate files from the same segment into a single Arena [lucene]

2024-07-23 Thread via GitHub
ChrisHegarty commented on code in PR #13570: URL: https://github.com/apache/lucene/pull/13570#discussion_r1687665071 ## lucene/core/src/java21/org/apache/lucene/store/RefCountedSharedArena.java: ## @@ -0,0 +1,138 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under

Re: [PR] Aggregate files from the same segment into a single Arena [lucene]

2024-07-23 Thread via GitHub
ChrisHegarty commented on code in PR #13570: URL: https://github.com/apache/lucene/pull/13570#discussion_r1687664305 ## lucene/core/src/java21/org/apache/lucene/store/RefCountedSharedArena.java: ## @@ -0,0 +1,138 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under

Re: [PR] Aggregate files from the same segment into a single Arena [lucene]

2024-07-22 Thread via GitHub
magibney commented on code in PR #13570: URL: https://github.com/apache/lucene/pull/13570#discussion_r1687033792 ## lucene/core/src/java21/org/apache/lucene/store/RefCountedSharedArena.java: ## @@ -0,0 +1,138 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one

Re: [PR] Aggregate files from the same segment into a single Arena [lucene]

2024-07-22 Thread via GitHub
dsmiley commented on code in PR #13570: URL: https://github.com/apache/lucene/pull/13570#discussion_r1686995275 ## lucene/core/src/java21/org/apache/lucene/store/RefCountedSharedArena.java: ## @@ -0,0 +1,138 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one o

Re: [PR] Aggregate files from the same segment into a single Arena [lucene]

2024-07-22 Thread via GitHub
magibney commented on code in PR #13570: URL: https://github.com/apache/lucene/pull/13570#discussion_r1686805197 ## lucene/core/src/java21/org/apache/lucene/store/RefCountedSharedArena.java: ## @@ -0,0 +1,138 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one

Re: [PR] Aggregate files from the same segment into a single Arena [lucene]

2024-07-20 Thread via GitHub
ChrisHegarty commented on PR #13570: URL: https://github.com/apache/lucene/pull/13570#issuecomment-2241152532 Subject to review comment, I think the only thing that remains is whether we want to allow the default number of total ref counting to be configurable, say by a system property or s

Re: [PR] Aggregate files from the same segment into a single Arena [lucene]

2024-07-20 Thread via GitHub
ChrisHegarty commented on code in PR #13570: URL: https://github.com/apache/lucene/pull/13570#discussion_r1685433634 ## lucene/core/src/java21/org/apache/lucene/store/RefCountedSharedArena.java: ## @@ -47,9 +48,12 @@ boolean acquire() { int value; while (true) {

Re: [PR] Aggregate files from the same segment into a single Arena [lucene]

2024-07-20 Thread via GitHub
ChrisHegarty commented on code in PR #13570: URL: https://github.com/apache/lucene/pull/13570#discussion_r1685433524 ## lucene/core/src/java21/org/apache/lucene/store/MemorySegmentIndexInputProvider.java: ## @@ -157,6 +157,8 @@ static Arena getSharedArena( arenas.comp

Re: [PR] Aggregate files from the same segment into a single Arena [lucene]

2024-07-19 Thread via GitHub
uschindler commented on code in PR #13570: URL: https://github.com/apache/lucene/pull/13570#discussion_r1684589983 ## lucene/core/src/java21/org/apache/lucene/store/MemorySegmentIndexInputProvider.java: ## @@ -157,6 +157,8 @@ static Arena getSharedArena( arenas.comput

Re: [PR] Aggregate files from the same segment into a single Arena [lucene]

2024-07-19 Thread via GitHub
uschindler commented on code in PR #13570: URL: https://github.com/apache/lucene/pull/13570#discussion_r1684583078 ## lucene/core/src/java21/org/apache/lucene/store/RefCountedSharedArena.java: ## @@ -47,9 +48,12 @@ boolean acquire() { int value; while (true) { v

Re: [PR] Aggregate files from the same segment into a single Arena [lucene]

2024-07-19 Thread via GitHub
ChrisHegarty commented on code in PR #13570: URL: https://github.com/apache/lucene/pull/13570#discussion_r1684456806 ## lucene/core/src/java21/org/apache/lucene/store/RefCountedSharedArena.java: ## @@ -47,9 +48,12 @@ boolean acquire() { int value; while (true) {

Re: [PR] Aggregate files from the same segment into a single Arena [lucene]

2024-07-19 Thread via GitHub
magibney commented on code in PR #13570: URL: https://github.com/apache/lucene/pull/13570#discussion_r1684426889 ## lucene/core/src/java21/org/apache/lucene/store/MemorySegmentIndexInputProvider.java: ## @@ -157,6 +157,8 @@ static Arena getSharedArena( arenas.computeI

Re: [PR] Aggregate files from the same segment into a single Arena [lucene]

2024-07-19 Thread via GitHub
magibney commented on code in PR #13570: URL: https://github.com/apache/lucene/pull/13570#discussion_r1684426889 ## lucene/core/src/java21/org/apache/lucene/store/MemorySegmentIndexInputProvider.java: ## @@ -157,6 +157,8 @@ static Arena getSharedArena( arenas.computeI

Re: [PR] Aggregate files from the same segment into a single Arena [lucene]

2024-07-19 Thread via GitHub
ChrisHegarty commented on code in PR #13570: URL: https://github.com/apache/lucene/pull/13570#discussion_r168408 ## lucene/core/src/java21/org/apache/lucene/store/RefCountedSharedArena.java: ## @@ -47,9 +48,12 @@ boolean acquire() { int value; while (true) {

Re: [PR] Aggregate files from the same segment into a single Arena [lucene]

2024-07-19 Thread via GitHub
ChrisHegarty commented on code in PR #13570: URL: https://github.com/apache/lucene/pull/13570#discussion_r168408 ## lucene/core/src/java21/org/apache/lucene/store/RefCountedSharedArena.java: ## @@ -47,9 +48,12 @@ boolean acquire() { int value; while (true) {

Re: [PR] Aggregate files from the same segment into a single Arena [lucene]

2024-07-19 Thread via GitHub
ChrisHegarty commented on code in PR #13570: URL: https://github.com/apache/lucene/pull/13570#discussion_r1684097085 ## lucene/core/src/java/org/apache/lucene/index/IndexFileNames.java: ## @@ -142,6 +143,26 @@ public static String stripSegmentName(String filename) { return

Re: [PR] Aggregate files from the same segment into a single Arena [lucene]

2024-07-18 Thread via GitHub
uschindler commented on code in PR #13570: URL: https://github.com/apache/lucene/pull/13570#discussion_r1683116076 ## lucene/core/src/java/org/apache/lucene/index/IndexFileNames.java: ## @@ -142,6 +143,26 @@ public static String stripSegmentName(String filename) { return fi

Re: [PR] Aggregate files from the same segment into a single Arena [lucene]

2024-07-18 Thread via GitHub
magibney commented on code in PR #13570: URL: https://github.com/apache/lucene/pull/13570#discussion_r1683132959 ## lucene/core/src/java/org/apache/lucene/index/IndexFileNames.java: ## @@ -142,6 +143,26 @@ public static String stripSegmentName(String filename) { return file

Re: [PR] Aggregate files from the same segment into a single Arena [lucene]

2024-07-18 Thread via GitHub
uschindler commented on code in PR #13570: URL: https://github.com/apache/lucene/pull/13570#discussion_r1683116076 ## lucene/core/src/java/org/apache/lucene/index/IndexFileNames.java: ## @@ -142,6 +143,26 @@ public static String stripSegmentName(String filename) { return fi

Re: [PR] Aggregate files from the same segment into a single Arena [lucene]

2024-07-18 Thread via GitHub
uschindler commented on code in PR #13570: URL: https://github.com/apache/lucene/pull/13570#discussion_r1683097237 ## lucene/core/src/java/org/apache/lucene/index/IndexFileNames.java: ## @@ -142,6 +143,26 @@ public static String stripSegmentName(String filename) { return fi

Re: [PR] Aggregate files from the same segment into a single Arena [lucene]

2024-07-18 Thread via GitHub
magibney commented on code in PR #13570: URL: https://github.com/apache/lucene/pull/13570#discussion_r1683080897 ## lucene/core/src/java/org/apache/lucene/index/IndexFileNames.java: ## @@ -142,6 +143,26 @@ public static String stripSegmentName(String filename) { return file

Re: [PR] Aggregate files from the same segment into a single Arena [lucene]

2024-07-18 Thread via GitHub
ChrisHegarty commented on code in PR #13570: URL: https://github.com/apache/lucene/pull/13570#discussion_r1683020894 ## lucene/core/src/java/org/apache/lucene/index/IndexFileNames.java: ## @@ -142,6 +143,26 @@ public static String stripSegmentName(String filename) { return

Re: [PR] Aggregate files from the same segment into a single Arena [lucene]

2024-07-18 Thread via GitHub
magibney commented on code in PR #13570: URL: https://github.com/apache/lucene/pull/13570#discussion_r1682897378 ## lucene/core/src/java/org/apache/lucene/index/IndexFileNames.java: ## @@ -142,6 +143,26 @@ public static String stripSegmentName(String filename) { return file

Re: [PR] Aggregate files from the same segment into a single Arena [lucene]

2024-07-18 Thread via GitHub
uschindler commented on code in PR #13570: URL: https://github.com/apache/lucene/pull/13570#discussion_r1682781560 ## lucene/core/src/java/org/apache/lucene/index/IndexFileNames.java: ## @@ -142,6 +143,26 @@ public static String stripSegmentName(String filename) { return fi

Re: [PR] Aggregate files from the same segment into a single Arena [lucene]

2024-07-18 Thread via GitHub
ChrisHegarty commented on code in PR #13570: URL: https://github.com/apache/lucene/pull/13570#discussion_r1682680606 ## lucene/core/src/java/org/apache/lucene/index/IndexFileNames.java: ## @@ -142,6 +143,26 @@ public static String stripSegmentName(String filename) { return

Re: [PR] Aggregate files from the same segment into a single Arena [lucene]

2024-07-18 Thread via GitHub
ChrisHegarty commented on code in PR #13570: URL: https://github.com/apache/lucene/pull/13570#discussion_r1682680606 ## lucene/core/src/java/org/apache/lucene/index/IndexFileNames.java: ## @@ -142,6 +143,26 @@ public static String stripSegmentName(String filename) { return

Re: [PR] Aggregate files from the same segment into a single Arena [lucene]

2024-07-18 Thread via GitHub
uschindler commented on code in PR #13570: URL: https://github.com/apache/lucene/pull/13570#discussion_r1682619481 ## lucene/core/src/java/org/apache/lucene/index/IndexFileNames.java: ## @@ -142,6 +143,26 @@ public static String stripSegmentName(String filename) { return fi

Re: [PR] Aggregate files from the same segment into a single Arena [lucene]

2024-07-18 Thread via GitHub
uschindler commented on code in PR #13570: URL: https://github.com/apache/lucene/pull/13570#discussion_r1682619481 ## lucene/core/src/java/org/apache/lucene/index/IndexFileNames.java: ## @@ -142,6 +143,26 @@ public static String stripSegmentName(String filename) { return fi

Re: [PR] Aggregate files from the same segment into a single Arena [lucene]

2024-07-18 Thread via GitHub
ChrisHegarty commented on code in PR #13570: URL: https://github.com/apache/lucene/pull/13570#discussion_r1682608032 ## lucene/core/src/java/org/apache/lucene/store/MMapDirectory.java: ## @@ -83,6 +86,19 @@ public class MMapDirectory extends FSDirectory { */ public static

Re: [PR] Aggregate files from the same segment into a single Arena [lucene]

2024-07-17 Thread via GitHub
uschindler commented on code in PR #13570: URL: https://github.com/apache/lucene/pull/13570#discussion_r1681847294 ## lucene/core/src/java/org/apache/lucene/store/MMapDirectory.java: ## @@ -83,6 +86,19 @@ public class MMapDirectory extends FSDirectory { */ public static f

Re: [PR] Aggregate files from the same segment into a single Arena [lucene]

2024-07-17 Thread via GitHub
uschindler commented on code in PR #13570: URL: https://github.com/apache/lucene/pull/13570#discussion_r1681841607 ## lucene/core/src/java/org/apache/lucene/store/MMapDirectory.java: ## @@ -167,6 +188,7 @@ public MMapDirectory(Path path, LockFactory lockFactory, long maxChunkSi

Re: [PR] Aggregate files from the same segment into a single Arena [lucene]

2024-07-17 Thread via GitHub
magibney commented on code in PR #13570: URL: https://github.com/apache/lucene/pull/13570#discussion_r1681711345 ## lucene/core/src/java/org/apache/lucene/store/MMapDirectory.java: ## @@ -83,6 +86,19 @@ public class MMapDirectory extends FSDirectory { */ public static fin

Re: [PR] Aggregate files from the same segment into a single Arena [lucene]

2024-07-17 Thread via GitHub
ChrisHegarty commented on PR #13570: URL: https://github.com/apache/lucene/pull/13570#issuecomment-2234130516 > So somebody what has more knowledge about how the filenames of deletion and/or softdeletes/docvalues instant update files are generated should help us. > > I think deleteion

Re: [PR] Aggregate files from the same segment into a single Arena [lucene]

2024-07-17 Thread via GitHub
ChrisHegarty commented on PR #13570: URL: https://github.com/apache/lucene/pull/13570#issuecomment-2234125875 I added a grouping function to the API - see [4591f7f](https://github.com/apache/lucene/pull/13570/commits/4591f7f904f0f76d4ef3fa5b4accee184183ddf1) -- This is an automated messag

Re: [PR] Aggregate files from the same segment into a single Arena [lucene]

2024-07-17 Thread via GitHub
uschindler commented on PR #13570: URL: https://github.com/apache/lucene/pull/13570#issuecomment-2233938475 So somebody what has more knowledge about how the filenames of deletion and/or softdeletes/docvalues instant update files are generated should help us. I think deleteion files s

Re: [PR] Aggregate files from the same segment into a single Arena [lucene]

2024-07-17 Thread via GitHub
uschindler commented on PR #13570: URL: https://github.com/apache/lucene/pull/13570#issuecomment-2233693234 Hi, this failure is unrelated. The difference is: Locally I use Windows 11, Jenkins uses Windows 10. Actually it looks like on Windows 11 it is in fact possible to delete memory mappe

Re: [PR] Aggregate files from the same segment into a single Arena [lucene]

2024-07-17 Thread via GitHub
uschindler commented on PR #13570: URL: https://github.com/apache/lucene/pull/13570#issuecomment-2233610476 Test also fails in main branch, so this is not new. Must have been to do with our previous changes. Interestingly it does not fail on [Policeman Jenkins](https://jenkins.thetaphi.de/j

Re: [PR] Aggregate files from the same segment into a single Arena [lucene]

2024-07-17 Thread via GitHub
uschindler commented on PR #13570: URL: https://github.com/apache/lucene/pull/13570#issuecomment-2233584894 > It fails reproducible. Let me look into it :-) Not sure what the issue is (if it is deletes, softdeletes or docvalues updates). This test failure is "interesting". This test e

Re: [PR] Aggregate files from the same segment into a single Arena [lucene]

2024-07-17 Thread via GitHub
ChrisHegarty commented on PR #13570: URL: https://github.com/apache/lucene/pull/13570#issuecomment-2233549425 > On Windows I got the following test error - actually what I expected (Windows is not allowed to delete files which are still open or mmapped Ha! you're too fast. I'm still s

Re: [PR] Aggregate files from the same segment into a single Arena [lucene]

2024-07-17 Thread via GitHub
uschindler commented on PR #13570: URL: https://github.com/apache/lucene/pull/13570#issuecomment-2233547375 It fails reproducible. Let me look into it :-) Not sure what the issue is (if it is deletes, softdeletes or docvalues updates). -- This is an automated message from the Apache Git S

Re: [PR] Aggregate files from the same segment into a single Arena [lucene]

2024-07-17 Thread via GitHub
ChrisHegarty commented on PR #13570: URL: https://github.com/apache/lucene/pull/13570#issuecomment-2233544810 > I am fine with everything - BUT: we should still check how the grouping works when for an existing segment an additional `*.del` file is updated (same for softdeletes and docvalue

Re: [PR] Aggregate files from the same segment into a single Arena [lucene]

2024-07-17 Thread via GitHub
uschindler commented on PR #13570: URL: https://github.com/apache/lucene/pull/13570#issuecomment-2233534218 On Windows I got the following test error - kindly what I expected (Windows is not allowed to delete files which are still open or mmapped): ``` TestIndexWriter > testDeleteU

Re: [PR] Aggregate files from the same segment into a single Arena [lucene]

2024-07-17 Thread via GitHub
uschindler commented on PR #13570: URL: https://github.com/apache/lucene/pull/13570#issuecomment-2233498077 I am fine with everything - BUT: we should still check how the grouping works when for an existing segment an additional `*.del` file is updated (same for softdeletes and docvalues up

Re: [PR] Aggregate files from the same segment into a single Arena [lucene]

2024-07-17 Thread via GitHub
magibney commented on PR #13570: URL: https://github.com/apache/lucene/pull/13570#issuecomment-2233474047 This is looking good. I think making it possible for MMapDirectory to customize/bypass grouping is still important -- something along the lines of what Uwe was thinking in [these](#issu

Re: [PR] Aggregate files from the same segment into a single Arena [lucene]

2024-07-17 Thread via GitHub
ChrisHegarty commented on code in PR #13570: URL: https://github.com/apache/lucene/pull/13570#discussion_r1681055574 ## lucene/core/src/java21/org/apache/lucene/store/MemorySegmentIndexInputProvider.java: ## @@ -125,4 +134,31 @@ private final MemorySegment[] map( } ret

Re: [PR] Aggregate files from the same segment into a single Arena [lucene]

2024-07-17 Thread via GitHub
ChrisHegarty commented on PR #13570: URL: https://github.com/apache/lucene/pull/13570#issuecomment-2233089190 > Can someone take a stab at summarizing this for CHANGES.txt, thus avoiding details a reader is unlikely to know about? Like me :-). How will a user of Lucene benefit? ```

Re: [PR] Aggregate files from the same segment into a single Arena [lucene]

2024-07-16 Thread via GitHub
uschindler commented on code in PR #13570: URL: https://github.com/apache/lucene/pull/13570#discussion_r1679766918 ## lucene/core/src/java21/org/apache/lucene/store/MemorySegmentIndexInputProvider.java: ## @@ -125,4 +134,31 @@ private final MemorySegment[] map( } retur

Re: [PR] Aggregate files from the same segment into a single Arena [lucene]

2024-07-16 Thread via GitHub
uschindler commented on code in PR #13570: URL: https://github.com/apache/lucene/pull/13570#discussion_r1679741852 ## lucene/core/src/java21/org/apache/lucene/store/MemorySegmentIndexInputProvider.java: ## @@ -125,4 +134,31 @@ private final MemorySegment[] map( } retur

Re: [PR] Aggregate files from the same segment into a single Arena [lucene]

2024-07-16 Thread via GitHub
uschindler commented on code in PR #13570: URL: https://github.com/apache/lucene/pull/13570#discussion_r1679741852 ## lucene/core/src/java21/org/apache/lucene/store/MemorySegmentIndexInputProvider.java: ## @@ -125,4 +134,31 @@ private final MemorySegment[] map( } retur

Re: [PR] Aggregate files from the same segment into a single Arena [lucene]

2024-07-16 Thread via GitHub
dsmiley commented on code in PR #13570: URL: https://github.com/apache/lucene/pull/13570#discussion_r1679708555 ## lucene/core/src/java21/org/apache/lucene/store/MemorySegmentIndexInputProvider.java: ## @@ -125,4 +134,31 @@ private final MemorySegment[] map( } return s

Re: [PR] Aggregate files from the same segment into a single Arena [lucene]

2024-07-16 Thread via GitHub
ChrisHegarty commented on code in PR #13570: URL: https://github.com/apache/lucene/pull/13570#discussion_r1679257863 ## lucene/core/src/java21/org/apache/lucene/store/MemorySegmentIndexInputProvider.java: ## @@ -125,4 +134,31 @@ private final MemorySegment[] map( } ret

Re: [PR] Aggregate files from the same segment into a single Arena [lucene]

2024-07-16 Thread via GitHub
ChrisHegarty commented on code in PR #13570: URL: https://github.com/apache/lucene/pull/13570#discussion_r1679175111 ## lucene/core/src/java/org/apache/lucene/store/MMapDirectory.java: ## @@ -199,20 +202,27 @@ public IndexInput openInput(String name, IOContext context) throws I

Re: [PR] Aggregate files from the same segment into a single Arena [lucene]

2024-07-16 Thread via GitHub
ChrisHegarty commented on code in PR #13570: URL: https://github.com/apache/lucene/pull/13570#discussion_r1679170017 ## lucene/core/src/java21/org/apache/lucene/store/MemorySegmentIndexInputProvider.java: ## @@ -125,4 +134,31 @@ private final MemorySegment[] map( } ret

Re: [PR] Aggregate files from the same segment into a single Arena [lucene]

2024-07-16 Thread via GitHub
ChrisHegarty commented on code in PR #13570: URL: https://github.com/apache/lucene/pull/13570#discussion_r1679168485 ## lucene/core/src/java21/org/apache/lucene/store/RefCountedSharedArena.java: ## @@ -0,0 +1,95 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under o

Re: [PR] Aggregate files from the same segment into a single Arena [lucene]

2024-07-16 Thread via GitHub
uschindler commented on code in PR #13570: URL: https://github.com/apache/lucene/pull/13570#discussion_r1678835899 ## lucene/core/src/java21/org/apache/lucene/store/MemorySegmentIndexInputProvider.java: ## @@ -125,4 +134,31 @@ private final MemorySegment[] map( } retur

Re: [PR] Aggregate files from the same segment into a single Arena [lucene]

2024-07-16 Thread via GitHub
uschindler commented on code in PR #13570: URL: https://github.com/apache/lucene/pull/13570#discussion_r1679007057 ## lucene/core/src/java/org/apache/lucene/store/MMapDirectory.java: ## @@ -199,20 +202,27 @@ public IndexInput openInput(String name, IOContext context) throws IOE

Re: [PR] Aggregate files from the same segment into a single Arena [lucene]

2024-07-16 Thread via GitHub
uschindler commented on code in PR #13570: URL: https://github.com/apache/lucene/pull/13570#discussion_r1679005565 ## lucene/core/src/java21/org/apache/lucene/store/MemorySegmentIndexInputProvider.java: ## @@ -125,4 +134,31 @@ private final MemorySegment[] map( } retur

Re: [PR] Aggregate files from the same segment into a single Arena [lucene]

2024-07-16 Thread via GitHub
uschindler commented on code in PR #13570: URL: https://github.com/apache/lucene/pull/13570#discussion_r1679005037 ## lucene/core/src/java21/org/apache/lucene/store/MemorySegmentIndexInputProvider.java: ## @@ -125,4 +134,31 @@ private final MemorySegment[] map( } retur

Re: [PR] Aggregate files from the same segment into a single Arena [lucene]

2024-07-15 Thread via GitHub
uschindler commented on code in PR #13570: URL: https://github.com/apache/lucene/pull/13570#discussion_r1678835899 ## lucene/core/src/java21/org/apache/lucene/store/MemorySegmentIndexInputProvider.java: ## @@ -125,4 +134,31 @@ private final MemorySegment[] map( } retur

Re: [PR] Aggregate files from the same segment into a single Arena [lucene]

2024-07-15 Thread via GitHub
uschindler commented on code in PR #13570: URL: https://github.com/apache/lucene/pull/13570#discussion_r1678835899 ## lucene/core/src/java21/org/apache/lucene/store/MemorySegmentIndexInputProvider.java: ## @@ -125,4 +134,31 @@ private final MemorySegment[] map( } retur

Re: [PR] Aggregate files from the same segment into a single Arena [lucene]

2024-07-15 Thread via GitHub
dsmiley commented on code in PR #13570: URL: https://github.com/apache/lucene/pull/13570#discussion_r1678459190 ## lucene/core/src/java21/org/apache/lucene/store/MemorySegmentIndexInputProvider.java: ## @@ -125,4 +134,31 @@ private final MemorySegment[] map( } return s

Re: [PR] Aggregate files from the same segment into a single Arena [lucene]

2024-07-15 Thread via GitHub
dsmiley commented on code in PR #13570: URL: https://github.com/apache/lucene/pull/13570#discussion_r1678448829 ## lucene/core/src/java/org/apache/lucene/store/MMapDirectory.java: ## @@ -199,20 +202,27 @@ public IndexInput openInput(String name, IOContext context) throws IOExce

Re: [PR] Aggregate files from the same segment into a single Arena [lucene]

2024-07-15 Thread via GitHub
magibney commented on PR #13570: URL: https://github.com/apache/lucene/pull/13570#issuecomment-2228833300 heh, was just writing to suggest something similar (`Function>`). SGTM. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

Re: [PR] Aggregate files from the same segment into a single Arena [lucene]

2024-07-15 Thread via GitHub
uschindler commented on PR #13570: URL: https://github.com/apache/lucene/pull/13570#issuecomment-2228809498 We may need some other logic: We have some files that can change without the segment name changing? How do we handle *.del od docValues update files? If they correspond to the same se

Re: [PR] Aggregate files from the same segment into a single Arena [lucene]

2024-07-15 Thread via GitHub
ChrisHegarty commented on code in PR #13570: URL: https://github.com/apache/lucene/pull/13570#discussion_r1678024394 ## lucene/core/src/java21/org/apache/lucene/store/RefCountedSharedArena.java: ## @@ -0,0 +1,102 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under

Re: [PR] Aggregate files from the same segment into a single Arena [lucene]

2024-07-15 Thread via GitHub
uschindler commented on PR #13570: URL: https://github.com/apache/lucene/pull/13570#issuecomment-2228783249 > To expand a bit on the concern I raised above: > > IIUC, in order for this to work properly (guaranteed to not potentially leak virtual memory address space) it depends on seg

Re: [PR] Aggregate files from the same segment into a single Arena [lucene]

2024-07-15 Thread via GitHub
magibney commented on PR #13570: URL: https://github.com/apache/lucene/pull/13570#issuecomment-2228755150 To expand a bit on the concern I raised above: IIUC, in order for this to work properly (guaranteed to not potentially leak virtual memory address space) it depends on segment fil

Re: [PR] Aggregate files from the same segment into a single Arena [lucene]

2024-07-15 Thread via GitHub
ChrisHegarty commented on code in PR #13570: URL: https://github.com/apache/lucene/pull/13570#discussion_r1677994244 ## lucene/core/src/java21/org/apache/lucene/store/RefCountedSharedArena.java: ## @@ -0,0 +1,102 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under

Re: [PR] Aggregate files from the same segment into a single Arena [lucene]

2024-07-15 Thread via GitHub
uschindler commented on PR #13570: URL: https://github.com/apache/lucene/pull/13570#issuecomment-2228727994 Hi, we also get Updates on the JDK fixes: https://github.com/openjdk/jdk/pull/20158 (see issue about more details and some improvements Solr and/or Elasticsearch/Opensearch should

Re: [PR] Aggregate files from the same segment into a single Arena [lucene]

2024-07-15 Thread via GitHub
uschindler commented on code in PR #13570: URL: https://github.com/apache/lucene/pull/13570#discussion_r1677959677 ## lucene/core/src/java21/org/apache/lucene/store/RefCountedSharedArena.java: ## @@ -0,0 +1,102 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under on

Re: [PR] Aggregate files from the same segment into a single Arena [lucene]

2024-07-15 Thread via GitHub
uschindler commented on code in PR #13570: URL: https://github.com/apache/lucene/pull/13570#discussion_r1677959677 ## lucene/core/src/java21/org/apache/lucene/store/RefCountedSharedArena.java: ## @@ -0,0 +1,102 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under on

Re: [PR] Aggregate files from the same segment into a single Arena [lucene]

2024-07-15 Thread via GitHub
uschindler commented on code in PR #13570: URL: https://github.com/apache/lucene/pull/13570#discussion_r1677946847 ## lucene/core/src/java21/org/apache/lucene/store/RefCountedSharedArena.java: ## @@ -0,0 +1,102 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under on

Re: [PR] Aggregate files from the same segment into a single Arena [lucene]

2024-07-15 Thread via GitHub
magibney commented on code in PR #13570: URL: https://github.com/apache/lucene/pull/13570#discussion_r1677916943 ## lucene/core/src/java21/org/apache/lucene/store/RefCountedSharedArena.java: ## @@ -0,0 +1,102 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one

Re: [PR] Aggregate files from the same segment into a single Arena [lucene]

2024-07-15 Thread via GitHub
ChrisHegarty commented on code in PR #13570: URL: https://github.com/apache/lucene/pull/13570#discussion_r1677637602 ## lucene/core/src/java21/org/apache/lucene/store/RefCountedSharedArena.java: ## @@ -0,0 +1,112 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under

Re: [PR] Aggregate files from the same segment into a single Arena [lucene]

2024-07-15 Thread via GitHub
ChrisHegarty commented on code in PR #13570: URL: https://github.com/apache/lucene/pull/13570#discussion_r1677620883 ## lucene/core/src/java/org/apache/lucene/store/MMapDirectory.java: ## @@ -199,20 +203,27 @@ public IndexInput openInput(String name, IOContext context) throws I

Re: [PR] Aggregate files from the same segment into a single Arena [lucene]

2024-07-15 Thread via GitHub
ChrisHegarty commented on code in PR #13570: URL: https://github.com/apache/lucene/pull/13570#discussion_r1677621308 ## lucene/core/src/java/org/apache/lucene/store/MMapDirectory.java: ## @@ -199,20 +203,27 @@ public IndexInput openInput(String name, IOContext context) throws I

Re: [PR] Aggregate files from the same segment into a single Arena [lucene]

2024-07-15 Thread via GitHub
uschindler commented on code in PR #13570: URL: https://github.com/apache/lucene/pull/13570#discussion_r1677544947 ## lucene/core/src/java/org/apache/lucene/store/MMapDirectory.java: ## @@ -199,20 +203,27 @@ public IndexInput openInput(String name, IOContext context) throws IOE

Re: [PR] Aggregate files from the same segment into a single Arena [lucene]

2024-07-15 Thread via GitHub
uschindler commented on code in PR #13570: URL: https://github.com/apache/lucene/pull/13570#discussion_r1677539871 ## lucene/core/src/java/org/apache/lucene/store/MMapDirectory.java: ## @@ -256,15 +267,15 @@ default IOException convertMapFailedIOException( } } - priva