Croway opened a new pull request, #30088: URL: https://github.com/apache/camel/pull/30088
[CAMEL-26998](https://issues.apache.org/jira/browse/CAMEL-26998) ## What was wrong `ScheduledPollConsumerHealthCheck` (the consumer health check of every `ScheduledPollConsumer`: file, ftp/sftp, sql, jpa, mail, aws2-s3, ...) reports from the state a poll leaves behind once it is done: the error counter and last error, which `ScheduledPollConsumer.doRun` updates after `poll()` returned or threw, and `isConsumerReady()`. A poll that does not return is invisible, so readiness stays UP for as long as it hangs. It is also UP when the first poll hangs with `camel.health.initialState=UP`, and when a consumer marked itself ready before its poll is done (`forceConsumerAsReady()`: sql after the query, ftp after the listing). Found with fault-injection tests (Toxiproxy blackhole / silent drop in front of a real PostgreSQL and SFTP server): camel-sql with pgjdbc's default `socketTimeout=0` waits in a socket read until TCP gives up or for ever; sftp with the default `soTimeout` (the JSch session timeout) hangs ~600 s. No error, no WARN, readiness UP. ## The fix - `ScheduledPollConsumer` times the most recent poll (a new `StopWatch` in a volatile field, set just before `pollingCounter` is incremented around begin/poll/commit), and has `getPollingDuration()`: the time since the poll in progress started, 0 when not polling. Each greedy run and each retry is a poll of its own. - New consumer option `healthCheckMaxPollDuration` on `ScheduledPollEndpoint` (so on every polling component), passed to the consumer. Default `0` = off. When set, the consumer health check is DOWN while a poll has been running longer than that, before any other rule (also before the first poll is done and for a consumer forced ready), with a `poll.duration` detail and the message: `Consumer has been polling for 10m5s, longer than healthCheckMaxPollDuration (10m0s) route: route1 (sftp://myserver/inbox)`. When the poll is done, the check is UP or DOWN by its result, as before. The poll is not interrupted. - Excluded from the AS2 and Box endpoints, which exclude the other scheduler options. - Docs: new section "Detecting a poll that hangs" in `health-check.adoc`, and a pointer from `batch-consumer.adoc`. The rest of the diff is the regenerated component JSON, configurers, URI factories, catalog and endpoint DSL for the new option. ## Design choices for reviewers - **Opt-in, no default.** A poll includes routing what it polled (a batch consumer routes the whole batch within `poll()`), so a big file, a large batch or a slow query is a long poll with nothing wrong. Any default (such as N × `delay`) would flip readiness for existing users; CAMEL-20189 shows how sensitive readiness during a long first poll is. No upgrade-guide entry, since nothing changes unless the option is set. - **Not a result threshold (CAMEL-17587).** CAMEL-17587 removed failure/success thresholds because a monitoring system can apply them to the results. Here there is no result to apply them to: while a poll hangs the check returns UP, and only Camel knows a poll is in progress. - **Per endpoint, not `camel.health.*`.** How long a poll may take depends on the endpoint (an SFTP route with big files vs. a SQL route), and it sits with the other scheduler options. A global default could come later if wanted. - **Unit:** milliseconds, typed as a duration (`10m` works), not affected by `timeUnit` (which the docs say is for `initialDelay`/`delay`). - **Concurrent polls** (`scheduler.concurrentConsumers` > 1): one watch, of the most recent poll to start, so the check is DOWN when polls are in progress and none has started for longer than the option; it never reports DOWN for a poll that has run less. Tracking the oldest of several polls would need a collection on every poll for a rare setup. - **Clock:** the duration is measured with `StopWatch`, which follows `TimeSource` (durations do; the timestamps of CAMEL-26715 stay on the system clock, so `ScheduledPollConsumer` does not call `TimeSource` itself and stays out of `TimeSourceDirectClockCallsTest`). The unit tests move a manual clock instead of waiting, and a deterministic simulation test gets it in virtual time. - Not done here: showing the polling duration in JMX / the consumer dev console. ## Tests - `ScheduledPollConsumerHangHealthCheckTest` (camel-core, manual clock, `@Isolated`): UP up to the max, DOWN at max + 1 ms with the message and detail, UP when the poll is done; a poll that fails after hanging is reported as a failed poll; off by default (1 h hang stays UP); a hanging first poll with each initial state; a consumer forced ready before its poll is done; each greedy run is a poll of its own; concurrent polls. - `FileConsumerHangHealthCheckTest` (camel-core): `file:...?healthCheckMaxPollDuration=200` through `ConsumerHealthCheck`, a route that blocks on a latch: UP, DOWN while blocked, UP after. - Fail before: on unmodified `main`, a consumer whose poll has been blocked for 2 s reports readiness UP with no message (`polling=true state=UP`). With the health-check part of this change removed (option and timing kept), 8 of the 9 unit tests fail (`expected: <DOWN> but was: <UP>`; the "off by default" one passes, as it should) and the file test times out waiting for DOWN. - `mvnd test -pl core/camel-core,core/camel-management -am -Dtest='*HealthCheck*Test,*ScheduledPoll*Test,TimeSource*Test,*PollingConsumer*Test,*Greedy*Test,*Backoff*Test,*PollEnrich*Test'`: camel-core 124, camel-management 5, camel-util 13 tests, all pass (including `TimeSourceDirectClockCallsTest`). - Generated files: the 74 changed component JSON files only gain the new option (checked by comparing them with the index fields removed); 69 endpoint DSL factories gain the two setters; AS2 and Box JSON are unchanged. _Claude Code on behalf of Croway_ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
