andygrove opened a new issue, #7079: URL: https://github.com/apache/datafusion-comet/issues/7079
### What is the problem the feature request solves? Completion tracking in the native Celeborn pusher polls on a timer and scans all pending pushes each time, and its transport hook installs a reflective proxy that affects every user of the Celeborn client in the executor. - `CelebornShufflePartitionPusher` schedules `reconcileAcceptedPushes` every `RECONCILIATION_INTERVAL_MILLIS = 10` ms on a shared scheduler while any push is pending. Each run holds `lifecycleLock` and walks every pending reservation and every observed push state, so the cost is O(pending) per tick per task even when nothing completed. - `CelebornTransportCallbackTracker` reflectively replaces the shared Celeborn transport factory's `clientBootstraps` list with one that includes a `java.lang.reflect.Proxy` hook, which wraps each client channel in a `TrackingChannel` proxy to observe completions. Every client built from that factory in the executor pays the invocation-handler hop on its channel calls, not only Comet's pushes. ### Describe the potential solution Complete reservations from the transport callbacks when they are available (the tracker already sees them) and keep the timer only as a low-frequency fallback; back the pending set with a structure that does not need a full scan per tick. Review whether the proxy can be scoped to the channels of Comet's own pushes, or replaced by a callback wrapper at the push call site. Weigh this against the ownership guarantees the tracker exists for (#5528). Measure with a multi-task executor pushing many small frames: scheduler CPU per second with the current 10 ms poll, and the added latency per channel call from the proxy. ### Additional context Finding C4 in #5905. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
