somandal opened a new pull request, #15973: URL: https://github.com/apache/pinot/pull/15973
Updating the upsert / dedup primary keys on segment add or replace can be an expensive resource intensive operation. This PR adds framework level support for throttling this operation via the same mechanism as `SegmentOperationsThrottler` for other segment operations such as segment download and pre-process. The following changes are included: - Modify the relevant upsert / dedup interfaces to receive a `SegmentOperationsThrottler` object as input. For now, pass this as null for both Concurrent map implementations of upserts and dedup - Add a new `SegmentHandleUpsertOrDedupThrottler` class to be used for performing the actual throttling via the use of a semaphore, whose number of permits can be updated dynamically via ZK cluster config updates. Wire this into `SegmentOperationsThrottler` - Add metrics to track the throttle threshold and the count of number of segments undergoing the operation - Update tests Testing done: - Tests updated and run locally - Manually verified via the Quickstart that config changes are picked up correctly Pending testing: - Need to test out the full logic with upserts / dedup to ensure that the semaphore is taken and released and only as many as the threshold allows -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
