CalvinKirs commented on PR #68197: URL: https://github.com/apache/doris/pull/68197#issuecomment-5728206430
Performance verification of this backport against a real PostgreSQL-backed Hive 2.3.2 metastore with 120,000 real partitions, driving the branch-4.1 production `ThriftHMSCachedClient` directly (pool size 1, warm connection, medians of repeated runs; identical result checksums for every before/after pair): | Partitions | Before RPCs | Before latency | After RPCs | After latency | Speedup | |---:|---:|---:|---:|---:|---:| | 1,000 | 1,000 | 5.702 s | 1 | 0.113 s | 50.5x | | 10,000 | 10,000 | 48.201 s | 2 | 0.536 s | 89.9x | | 120,000 | 120,000 | 625.284 s | 24 | 6.509 s | 96.1x | *Before* is the per-partition access shape of the base branch (one `getPartitionsByNames` RPC per name — cold per-key cache loads and per-partition MTMV freshness reads). *After* is one logical call chunked at the default batch size of 5,000. Batch-size sensitivity at 120,000 partitions (single runs, exercising the `hive.hms_partitions_batch_size_per_rpc` catalog property end to end): 100 → 12.9 s, 500 → 7.5 s, 1,000 → 8.1 s, 2,500 → 7.2 s, 5,000 → 6.6 s. Notably, the base branch's *unbounded* bulk shape (all 120,000 names in one `getPartitionsByNames`, the physical shape of `HiveExternalMetaCache.loadPartitions` before this change) failed outright in the same session: the metastore worker thread died with `java.lang.StackOverflowError` planning the 120,000-element name list and the client saw a `TTransportException`, while the chunked path read the same partitions in 6.2–6.7 s. At this scale bounded chunking is the difference between completing and crashing the metastore request. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
