gavinchou commented on code in PR #65788:
URL: https://github.com/apache/doris/pull/65788#discussion_r4282480040
##########
be/src/cloud/cloud_storage_engine.cpp:
##########
@@ -103,6 +103,54 @@ CloudStorageEngine::CloudStorageEngine(const
EngineOptions& options)
_cumulative_compaction_policies[CUMULATIVE_TIME_SERIES_POLICY] =
std::make_shared<CloudTimeSeriesCumulativeCompactionPolicy>();
_startup_timepoint = std::chrono::system_clock::now();
+ // Build the dedicated, bounded query-cache delta pre-sync pool up front
(see
+ // the member declaration): a decision fan-out must never find it null,
and the
+ // destructor's stop() drains it before _meta_mgr/_tablet_mgr die. A build
+ // failure here means the process cannot create threads at startup, already
+ // fatal, so CHECK rather than limp on with a null pool.
+ //
+ // Report a misconfigured sizing loudly instead of silently coercing it: an
+ // operator who set an invalid value must see why the BE refused to start
rather
+ // than run with a capacity that differs from the config (invalid state
fails, it
+ // never silently continues). A max of >= 1 is required (a zero-width pool
can run
+ // nothing) and the queue bound must be non-negative.
+ int32_t qc_sync_threads = config::query_cache_delta_sync_thread;
+ int32_t qc_sync_queue = config::query_cache_delta_sync_max_pending_tasks;
+ CHECK_GE(qc_sync_threads, 1) << "query_cache_delta_sync_thread must be >=
1, got "
+ << qc_sync_threads;
+ CHECK_GE(qc_sync_queue, 0) << "query_cache_delta_sync_max_pending_tasks
must be >= 0, got "
+ << qc_sync_queue;
+ // Pre-start all workers here (min == max), at engine construction, OFF
the query
+ // admission path. The decision fan-out submits from operator init, which
runs on
+ // the bounded light_work_pool, and ThreadPool creates workers
synchronously
+ // inside submit_func when the pool is short of the demand (creation can
take
+ // hundreds of ms). A lazily grown pool (min = 0) would move that creation
latency
+ // onto the query's critical path and, worse, could hand the fan-out a
submit that
+ // enqueued a task yet returned an error at zero live threads. A fixed
pool sized
+ // up front keeps submit_func pure enqueue (it never creates a thread, so
it never
+ // blocks), so the wait is spent on the sync RPCs rather than on
+ // spawning threads. The cost is a fixed set of mostly-idle workers on
every cloud
+ // BE even when this opt-in feature is unused; that steady footprint is the
+ // deliberate trade for predictable, non-blocking admission.
+ Status qc_pool_st = ThreadPoolBuilder("QueryCacheDeltaSyncThreadPool")
Review Comment:
Could you please clarify the specific differences between the thread pool
handling synchronized data and the scanner's synchronized data? We are
exploring whether these two logical components can be integrated, especially in
the context of overall concurrency control.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
