github-actions[bot] commented on code in PR #69422:
URL: https://github.com/apache/doris/pull/69422#discussion_r4374287393
##########
be/src/runtime/exec_env.cpp:
##########
@@ -191,20 +194,44 @@ std::map<TNetworkAddress, FrontendInfo>
ExecEnv::get_running_frontends() {
return res;
}
+void ExecEnv::wait_for_shutdown_heartbeat() {
+ int32_t elapsed = 0;
+ while (!k_shutdown_heartbeat_sent && elapsed <
config::grace_shutdown_heartbeat_wait_seconds) {
+ sleep(1);
+ ++elapsed;
+ }
+ LOG(INFO) << "shutdown heartbeat prepared: " <<
k_shutdown_heartbeat_sent.load() << ", waited "
+ << elapsed << " seconds";
+ // The response may still be in transit. This buffer also gives followers
time
+ // to replay shutdown state. It does not prove that every FE has applied
it.
+ sleep(std::max(0, config::grace_shutdown_heartbeat_propagation_seconds));
+}
+
void ExecEnv::wait_for_all_tasks_done() {
- // For graceful shutdown, need to wait for all running queries to stop
+ // Both receiver implementations and the load coordinator must finish. For
example,
+ // an HTTP upload or transaction commit can outlive all of its pipeline
fragments.
int32_t wait_seconds_passed = 0;
while (true) {
int num_queries = _fragment_mgr->running_query_num();
- if (num_queries < 1) {
+ size_t num_channels = _load_channel_mgr->num_active_channels();
+ // TODO: Fix partial-open receiver cleanup (see
LoadStream::on_closed()) so
+ // abandoned entries do not consume the full
grace_shutdown_wait_seconds.
+ // Keep counting receivers until then to avoid exiting during active
loads.
+ size_t num_streams = _load_stream_mgr->get_load_stream_num();
+ size_t num_loads = StreamLoadContext::num_active_contexts();
+ if (num_queries == 0 && num_channels == 0 && num_streams == 0 &&
num_loads == 0) {
Review Comment:
[P2] Count accepted receiver opens before declaring shutdown idle
An accepted `PInternalService::open_load_stream()` RPC is not counted until
after its tablet lookups and `LoadStreamMgr` registration. In the supported
shared-nothing V2 path, `ExecEnv::get_tablet()` can block on the tablet-shard
shared lock: a concurrent CREATE_TABLE for another tablet in that shard holds
the exclusive lock across filesystem `exists`/`create_directory` calls
(`TabletManager::create_tablet()`). If this BE only receives the load and its
coordinator runs elsewhere, all four counts here remain zero during that wait.
With a prompt shutdown heartbeat, a filesystem stall longer than the
propagation/post-delay allowances but shorter than the sender's 60-second open
timeout can therefore cause this BE to exit without using its 120-second
work-drain budget, failing a load that could still finish. Reserve accepted
opens before queuing/blocking setup and retain that reservation until
registration takes over or the open fails. This differs from the existing
partial-open threa
d, which concerns an overcount after registration.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
