allthingssecurity opened a new pull request, #30459:
URL: https://github.com/apache/camel/pull/30459

   # Description
   
   [CAMEL-27212](https://issues.apache.org/jira/browse/CAMEL-27212)
   
   Two problems with the thread pool of the Circuit Breaker EIP 
(camel-resilience4j) when the timeout is enabled:
   
   - Asynchronous mode (`asynchronous=true`): the reifier creates one 
`ScheduledThreadPoolExecutor` (or takes the `timeoutExecutorService`) and the 
processor uses it both to run the calls and as the scheduler of the 
`TimeLimiter`, so the timeout of a call is a task of the same pool. When the 
calls block their threads (a synchronous http call to a server that does not 
answer, which is the case the timeout is for) and there are as many of them as 
threads in the pool (10 with the default profile), no thread is left to run the 
timeouts: no call times out, no fallback runs, the exchanges wait for as long 
as the server does not answer, and the circuit breaker never records the 
failures, so it does not open. The docs example 
(`asynchronous(true).timeoutEnabled(true).timeoutDuration(2000)` around an 
`http:` call) runs into it under load. `ResilienceAsyncTimeoutTest` uses 
`delay()`, which gives the thread back, so it does not show it.
   - Route restart (any mode): the reifier creates the thread pool once, with 
the route, but `ResilienceProcessor.doStop` shuts it down. After `stopRoute` + 
`startRoute` (JMX, supervising route controller, CLI) the pool is terminated. 
The rejection policy of the pool (CallerRuns) discards a task once the pool is 
shut down, without an exception: in synchronous mode every call times out 
without running and gets the fallback (and the circuit breaker opens); in 
asynchronous mode the call and its timeout are both discarded, so the exchange 
never completes and the caller waits for ever. It stays so until the route is 
removed and added again.
   
   Fix:
   
   - In asynchronous mode the processor schedules the timeouts on threads of 
its own: a scheduled thread pool (default profile) created when the processor 
starts and shut down when the route is removed. The thread pool of the reifier 
(or `timeoutExecutorService`) runs only the calls. A `scheduledExecutorService` 
set on the processor by code is still used for the timeouts. The fallback, and 
the rest of the route, run on the timeout thread when the call times out, as 
they ran on a thread of the shared pool before (one more sentence in the docs).
   - The thread pools are shut down in `doShutdown` (route removal, 
CamelContext stop) instead of `doStop`, as the other EIPs with their own thread 
pool do (Threads, Multicast, WireTap).
   
   Model: a small TLA+ model (`R4jAsyncTimeout.tla`: two calls that hang, a 
pool of one thread, the timeouts in the same pool or on their own threads, a 
route stop + start) checks "every call gets an answer" and "a call is not 
answered without running because of a restart". On main TLC reports a deadlock 
in 4 states (the call blocks the only thread, its timeout waits behind it) and 
the restart invariant is violated in 3 states; with the change both hold (32 
and 64 distinct states).
   
   Tests:
   
   - `ResilienceAsyncTimeoutBlockedPoolTest`: asynchronous mode, timeout 200 
ms, a `timeoutExecutorService` with one thread, a step that blocks: the 
fallback must come within 5 seconds.
   - `ResilienceRouteRestartTest`: synchronous and asynchronous mode with a 
timeout; after `stopRoute` + `startRoute` the steps of the circuit breaker must 
still run.
   
   On main all three fail: no fallback within 5 seconds (blocked pool); after 
the restart the synchronous route answers `Fallback` instead of `Bye World`, 
and the asynchronous one never answers. With the change they pass, and the 
camel-resilience4j tests pass (101 run, 0 failures, built with camel-management 
of main for `ResilienceFallbackManagementTest`). The two new classes 20 times 
in one JVM: 60 run, 0 failures.
   
   Not changed: the reifier still requires a `ScheduledExecutorService` for 
`timeoutExecutorService` in asynchronous mode (it now only runs the calls, so 
the check could be relaxed in a follow-up). camel-microprofile-fault-tolerance 
is not affected by the first problem (SmallRye uses its own timer), and its 
synchronous guard does not use the thread pool that its `doStop` shuts down.
   
   # Target
   
   - [x] I checked that the commit is targeting the correct branch (Camel 4 
uses the `main` branch)
   
   # Tracking
   - [x] If this is a large change, bug fix, or code improvement, I checked 
there is a [JIRA issue](https://issues.apache.org/jira/browse/CAMEL) filed for 
the change (usually before you start working on it): 
[CAMEL-27212](https://issues.apache.org/jira/browse/CAMEL-27212).
   
   # Apache Camel coding standards and style
   
   - [x] I checked that each commit in the pull request has a meaningful 
subject line and body.
   - [ ] I have run `mvn clean install -DskipTests` locally from root folder 
and I have committed all auto-generated changes.
     (I built the touched modules with the formatter and import-sort plugins; 
the docs change is in the EIP page and its catalog copy. I did not run the full 
root build.)
   
   # AI-assisted contributions
   
   - [x] If this PR includes AI-generated code, commits have proper 
co-authorship attribution (e.g., `Co-authored-by` trailers) and the PR 
description identifies the AI tool used.
     This PR was prepared with Claude Code (Claude Opus 5.5). The commit 
carries a `Co-Authored-By` trailer.
   
   _Claude Code on behalf of allthingssecurity_
   
   🤖 Generated with [Claude Code](https://claude.com/claude-code)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to