dinoocch opened a new pull request, #16122: URL: https://github.com/apache/pinot/pull/16122
Previously this class stored a map from `segmentName -> partitionDetails`. This works but means that during prune() we need to check to see if the partition is matching for each and every segment. <img width="1247" alt="Screenshot 2025-06-12 at 2 22 17 PM" src="https://github.com/user-attachments/assets/047e3747-fec7-4fbd-a4df-716f44a957f2"; /> In this case ~40% of the total broker cpu time is spent in `SinglePartitionColumnSegmentPruner::prune()`. This change observes that generally the total number of partitions will be much smaller than the number of segments, so instead of mapping segment to partition config, invert this pattern and map a partition to a set of segments. There's some interesting behavior to consider I think -- for example, a segment in both partitions 1 and 5 would be present in both the set for 1 and the set for 5. In the "best" case, each segment participates in exactly 1 partition. This means that for equality we have to do a single set intersection with the input `Set<String> segments | Set<String> matchingPartition`. * `numPartition` partition evaluations (note this can temporarily be higher than the number of partitions, for example if the number of partitions changed and a backfill/refresh was not done to update the metadata). * One set intersection, comparing `matchingPartition` with `segments`. * Generally `matchingPartition` size will be `1 / partitionCount * numSegments` In the "worst" common case, consider a filter that matches every partition -- this will still do `numPartition` evaluations, but will also have to do more set intersections. Finally, the "worst" case would be a very badly configured partitioning in which segments participate in every partition. We would still do `numPartition` evaluations and have set intersections done on each set of segments, but in this case there would be `numSegments` in each `Set` and `partition` sets => `numSegments * numPartitions` calls to `segments.contains`... Will write some simple benchmarks to compare performance of `prune()`. I also wonder if we would see benefit from maintaining some setup like `Map<PartitionInfo, BloomFilter<String>>` instead of the exact pruning. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
