Re: [PR] Reduce memory usage of SkipListWriter [lucene]

2024-07-29 Thread via GitHub
bugmakerr commented on PR #13576: URL: https://github.com/apache/lucene/pull/13576#issuecomment-2257386376 @jpountz @mikemccand could you take a look when you get a chance -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub an

Re: [PR] WIP: draft of intra segment concurrency [lucene]

2024-07-29 Thread via GitHub
javanna commented on PR #13542: URL: https://github.com/apache/lucene/pull/13542#issuecomment-2257058937 Tests are now green! I marked this ready for review. I have addressed the remaining testing issues, fixed `FacetsCollectorManager` to support intra-segment concurrency transparentl

Re: [PR] WIP: draft of intra segment concurrency [lucene]

2024-07-29 Thread via GitHub
javanna commented on code in PR #13542: URL: https://github.com/apache/lucene/pull/13542#discussion_r1695912482 ## lucene/facet/src/test/org/apache/lucene/facet/TestDrillSideways.java: ## @@ -146,114 +146,123 @@ private IndexSearcher getNewSearcher(IndexReader reader) { //

Re: [PR] WIP: draft of intra segment concurrency [lucene]

2024-07-29 Thread via GitHub
javanna commented on code in PR #13542: URL: https://github.com/apache/lucene/pull/13542#discussion_r1695859578 ## lucene/core/src/test/org/apache/lucene/search/TestSortRandom.java: ## @@ -119,7 +119,8 @@ private void testRandomStringSort(SortField.Type type) throws Exception {

Re: [PR] WIP: draft of intra segment concurrency [lucene]

2024-07-29 Thread via GitHub
javanna commented on code in PR #13542: URL: https://github.com/apache/lucene/pull/13542#discussion_r1695857209 ## lucene/core/src/java/org/apache/lucene/search/PointRangeQuery.java: ## @@ -362,6 +362,9 @@ public long cost() { final IntersectVisitor visitor = getInt

Re: [PR] Deduplicate bytes for `FieldReader#rootCode` [lucene]

2024-07-29 Thread via GitHub
original-brownbear commented on PR #13610: URL: https://github.com/apache/lucene/pull/13610#issuecomment-2256450538 @mikemccand It's admittedly the fault of design decisions made and each individually isn't that big. We've simply navigated ourselves into a situation where we have far too ma

Re: [PR] Deduplicate bytes for `FieldReader#rootCode` [lucene]

2024-07-29 Thread via GitHub
mikemccand commented on PR #13610: URL: https://github.com/apache/lucene/pull/13610#issuecomment-2256424690 Egads, how does this add up to 100s of MB? Is each `rootCode` really so big? I think they are indeed one and the same. I'm not sure why we are duplicating it -- is it really d

Re: [PR] Compute facets while collecting [lucene]

2024-07-29 Thread via GitHub
gsmiller commented on code in PR #13568: URL: https://github.com/apache/lucene/pull/13568#discussion_r1695329026 ## lucene/sandbox/src/java/org/apache/lucene/sandbox/facet/FacetFieldCollectorManager.java: ## @@ -0,0 +1,55 @@ +/* + * Licensed to the Apache Software Foundation (AS

Re: [PR] Compute facets while collecting [lucene]

2024-07-29 Thread via GitHub
gsmiller commented on code in PR #13568: URL: https://github.com/apache/lucene/pull/13568#discussion_r1695319115 ## lucene/sandbox/src/java/org/apache/lucene/sandbox/facet/ordinals/OrdinalGetter.java: ## @@ -0,0 +1,24 @@ +/* + * Licensed to the Apache Software Foundation (ASF) u

Re: [PR] Compute facets while collecting [lucene]

2024-07-29 Thread via GitHub
gsmiller commented on code in PR #13568: URL: https://github.com/apache/lucene/pull/13568#discussion_r1695316438 ## lucene/sandbox/src/java/org/apache/lucene/sandbox/facet/ordinals/CandidateSetOrdinalIterator.java: ## @@ -0,0 +1,48 @@ +/* + * Licensed to the Apache Software Foun

Re: [PR] Compute facets while collecting [lucene]

2024-07-29 Thread via GitHub
gsmiller commented on code in PR #13568: URL: https://github.com/apache/lucene/pull/13568#discussion_r1695312400 ## lucene/sandbox/src/java/org/apache/lucene/sandbox/facet/recorders/FacetRecorder.java: ## @@ -0,0 +1,55 @@ +/* + * Licensed to the Apache Software Foundation (ASF)

Re: [PR] Use Max WAND optimizations with ToParentBlockJoinQuery when using ScoreMode.Max [lucene]

2024-07-29 Thread via GitHub
Mikep86 commented on code in PR #13587: URL: https://github.com/apache/lucene/pull/13587#discussion_r1695262457 ## lucene/join/src/test/org/apache/lucene/search/join/TestBlockJoinScorer.java: ## @@ -118,4 +110,114 @@ public void testScoreNone() throws IOException { reader.c

Re: [PR] Use Max WAND optimizations with ToParentBlockJoinQuery when using ScoreMode.Max [lucene]

2024-07-29 Thread via GitHub
Mikep86 commented on code in PR #13587: URL: https://github.com/apache/lucene/pull/13587#discussion_r1695259920 ## lucene/join/src/test/org/apache/lucene/search/join/TestBlockJoinScorer.java: ## @@ -16,23 +16,16 @@ */ package org.apache.lucene.search.join; +import static or

Re: [PR] Deduplicate bytes for `FieldReader#rootCode` [lucene]

2024-07-29 Thread via GitHub
jpountz commented on PR #13610: URL: https://github.com/apache/lucene/pull/13610#issuecomment-2255773311 The write logic computes the `rootCode` as `root.index.getEmptyOutput()` so it looks like you may be right. @mikemccand Do you know a bit more about this? -- This is an automated messa

Re: [PR] Compute facets while collecting [lucene]

2024-07-29 Thread via GitHub
epotyom commented on code in PR #13568: URL: https://github.com/apache/lucene/pull/13568#discussion_r1695011732 ## lucene/facet/src/java/org/apache/lucene/facet/DrillSideways.java: ## @@ -398,130 +490,56 @@ private ConcurrentDrillSidewaysResult searchSequentially( }

[PR] Removing all deprecated TopScoreDocCollector + TopFieldCollector methods (#create, #createSharedManager) [lucene]

2024-07-29 Thread via GitHub
slow-J opened a new pull request, #13617: URL: https://github.com/apache/lucene/pull/13617 Following up on: https://github.com/apache/lucene/pull/13500 These methods already marked for deprecation in 9.x and we previously removed all internal use of these methods in 10.0 (in https://

Re: [PR] Compute facets while collecting [lucene]

2024-07-29 Thread via GitHub
epotyom commented on code in PR #13568: URL: https://github.com/apache/lucene/pull/13568#discussion_r1694979476 ## lucene/sandbox/src/java/org/apache/lucene/sandbox/facet/cutters/ranges/LongRangeFacetCutter.java: ## @@ -0,0 +1,413 @@ +/* + * Licensed to the Apache Software Found

Re: [PR] Removing usage of TopScoreDocCollector + TopFieldCollector deprecated methods (#create, #createSharedManager) [lucene]

2024-07-29 Thread via GitHub
slow-J commented on PR #13500: URL: https://github.com/apache/lucene/pull/13500#issuecomment-2255548859 Thanks @javanna for the review! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specifi

Re: [PR] Release memory for cancelled tasks earlier in TaskExecutor [lucene]

2024-07-29 Thread via GitHub
original-brownbear commented on PR #13609: URL: https://github.com/apache/lucene/pull/13609#issuecomment-2255530896 > I believe this change is ok, but I am no longer very confident about TestTaskExecutor#testCancelTasksOnException testing a real-life scenario, because it was recently change

Re: [PR] Release memory for cancelled tasks earlier in TaskExecutor [lucene]

2024-07-29 Thread via GitHub
javanna commented on PR #13609: URL: https://github.com/apache/lucene/pull/13609#issuecomment-2255481310 I believe this change is ok, but I am no longer very confident about `TestTaskExecutor#testCancelTasksOnException` testing a real-life scenario, because it was recently changed to no lon

Re: [PR] Compute facets while collecting [lucene]

2024-07-29 Thread via GitHub
epotyom commented on code in PR #13568: URL: https://github.com/apache/lucene/pull/13568#discussion_r1694907952 ## lucene/sandbox/src/java/org/apache/lucene/sandbox/facet/FacetFieldCollectorManager.java: ## @@ -0,0 +1,55 @@ +/* + * Licensed to the Apache Software Foundation (ASF

Re: [PR] Removing usage of TopScoreDocCollector + TopFieldCollector deprecated methods (#create, #createSharedManager) [lucene]

2024-07-29 Thread via GitHub
javanna commented on PR #13500: URL: https://github.com/apache/lucene/pull/13500#issuecomment-2255396796 Thanks @slow-J ! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

Re: [PR] Removing usage of TopScoreDocCollector + TopFieldCollector deprecated methods (#create, #createSharedManager) [lucene]

2024-07-29 Thread via GitHub
javanna merged PR #13500: URL: https://github.com/apache/lucene/pull/13500 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucene.apa

Re: [PR] Add WrappedCandidateMatcher for composing matchers [lucene]

2024-07-29 Thread via GitHub
romseygeek commented on PR #13109: URL: https://github.com/apache/lucene/pull/13109#issuecomment-2255366931 Thanks, that's very helpful. Given that composite matchers need to call those methods for CandidateMatcher to be useful, I think the simplest way forward here is to make `matchQuery(

Re: [PR] Compute facets while collecting [lucene]

2024-07-29 Thread via GitHub
epotyom commented on code in PR #13568: URL: https://github.com/apache/lucene/pull/13568#discussion_r1694819821 ## lucene/sandbox/src/java/org/apache/lucene/sandbox/facet/misc/LongValueFacetCutter.java: ## @@ -0,0 +1,155 @@ +/* + * Licensed to the Apache Software Foundation (ASF