xiangfu0 opened a new pull request, #18992: URL: https://github.com/apache/pinot/pull/18992
## Summary - Add an optional `segmentCompressedSizeBytes` completion-protocol parameter so realtime servers can report the generated tar.gz size separately from the unpacked build size. - Persist the compressed archive size as `segment.size.in.bytes`, while retaining the unpacked size for flush auto-tuning and as a rolling-upgrade fallback. - Use the archive size in deep-store recovery, refresh stale DONE metadata after a successful recovery upload, and label the controller UI value as **Compressed Segment Size**. - Document the size-field semantics, rollout behavior, and sample API queries in `pinot-tools/src/main/resources/examples/compression-stats/README.md`. ## Motivation Realtime segment completion previously used `segment.getSegmentSizeBytes()`, which measures the generated unpacked segment directory, for `segment.size.in.bytes`. Other segment ingestion paths define that metadata field as the downloadable compressed archive size. This made realtime metadata substantially larger than the persisted tar.gz and caused confusing comparisons with server disk-size APIs. ## Compatibility The new protocol parameter is optional. During rolling upgrades: - A new controller receiving a commit from an old server falls back to the existing unpacked `segmentSizeBytes` value. - An old controller ignores the extra parameter from a new server and preserves the prior behavior. - Flush auto-tuning continues to consume the unpacked build size, so row-limit behavior is unchanged. - Existing metadata is not proactively rewritten. An upgraded deep-store recovery can refresh the value when it rebuilds the archive. ## User guide and sample queries The updated compression-statistics example guide explains the difference between `segment.size.in.bytes`, verbose table-size `diskSizeInBytes`, and compression-statistics value sizes. It also includes runnable `curl` and `jq` queries for comparing one realtime segment across the metadata and table-size APIs. ## Testing - `SegmentCompletionProtocolTest` - `RealtimeSegmentDataManagerTest` (27 tests) - `PinotLLCRealtimeSegmentManagerTest` and `SizeBasedSegmentFlushThresholdComputerTest` (1,070 tests) - `TablesResourceTest` (21 tests) - Controller UI Vite production bundle build - `spotless:apply`, `license:format`, `checkstyle:check`, and `license:check` for `pinot-common`, `pinot-core`, `pinot-server`, and `pinot-controller` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
