allthingssecurity opened a new pull request, #30132: URL: https://github.com/apache/camel/pull/30132
# Description [CAMEL-26967](https://issues.apache.org/jira/browse/CAMEL-26967) Found by the Kafka DST world of #29992 (CAMEL-26947), which leaves this case out of its default set: with `-Ddst.kafka.syncBreakOnFirstError=true`, seed 95 and 5 more of 1000 seeds commit an offset past a record the route never finished. Reproduced here with a plain unit test, outside the DST harness. With `breakOnFirstError=true` and synchronous manual commits (`allowManualCommit=true` with `DefaultKafkaManualCommitFactory`, so the `SyncCommitManager`), a failed record makes `KafkaRecordStreamingProcessor.processException` commit the records routed before it and return the break; `KafkaFetchRecords` then reconnects, and the new Kafka consumer polls again from the committed offset. That commit is a `commitSync` with `commitTimeoutMs`, and it can throw: `TimeoutException` when the timeout expires first (the caller does not know whether the commit was applied), or `CommitFailedException` / `RebalanceInProgressException`. The exception left the batch and reached the catch of `KafkaFetchRecords.startPolling`, and with `pollOnError` `ERROR_HANDLER` (the default), `DISCARD` or `RETRY` the fetcher went on polling with the same Kafka consumer. Its position was already after the whole poll, so the failed record and the records after it in that poll were never routed, and the next manual commit of a later record committed past them. The option documentation says that with the sync or async commit manager "the consumer will seek back to the offset of the message that caused a failure, and then re-attempt to process this message", and `KafkaConsumer#commitSync(Map, Duration)` documents the `TimeoutException` as one of its outcomes. Change, in `KafkaRecordStreamingProcessor.processException`: a `KafkaException` of that commit is reported to the exception handler of the consumer and the break is returned as before, so the consumer reconnects and polls from the committed offset. If the commit was applied, that is the failed record; if not, an earlier offset, and some records are routed again (at-least-once) instead of skipped. `WakeupException` and `InterruptException` still go to the poll loop as before. Visible change: that commit failure no longer goes through `pollOnError`. It reaches the exception handler of the consumer (logged by default, routed with `bridgeErrorHandler=true`) instead of the error handler of the route through the `ERROR_HANDLER` strategy, and with `pollOnError=STOP` the consumer reconnects (as the break does) instead of stopping. Both only for a failed commit in the break. Not changed: the commit of `KafkaConsumerListener.afterProcess` (the resumable consumer with a resume strategy) can fail the same way and still goes through `pollOnError`; I can open a follow-up JIRA. The commit at the end of a partition runs only without manual commits, where the commit manager makes no Kafka commit (no-op, or the offset repository). The batching consumer is not affected: its break calls `commitManager.commit()`, which does nothing with manual commits, and with auto-commit the commit runs in an on-completion. Test, new `KafkaConsumerBreakOnFirstErrorCommitTimeoutTest` (a `MockConsumer` on a log, a new Kafka consumer starts from the committed offset of the group, as in `KafkaConsumerBreakOnFirstErrorPausedTest`): records a to e, the route commits each record manually, c fails once, the `commitSync` of the break times out (not applied), and f is stored after it. For `pollOnError` `ERROR_HANDLER`, `DISCARD` and `RETRY` the route must get a to f in order; `RECONNECT` is a control that passes without the change too. `testFailedCommitOfTheBreakIsReportedToTheExceptionHandler` checks the report (with `RECONNECT`, which polls the record again on main too). Without the change the three cases and the report test fail (the route gets `[a, b, f]` and the group commits offset 6; nothing reaches the exception handler); with it all five pass, and the camel-kafka unit tests pass (340 run, 0 failures, 1 skipped). With the change merged locally into the world of #29992, `-Ddst.kafka.syncBreakOnFirstError=true` passes 1000 of 1000 seeds (6 fail without it), and the default set passes 1000 of 1000. The ITs need a Kafka container and were not run locally (CI runs them). The local offline build used the cached kafka-clients 4.3.1. No documentation or upgrade guide change: the consumer now does what the option already describes. Backport: camel-4.18.x has the same `processException`; the change applies there as is (records are lost, so it may be worth it). # Target - [x] I checked that the commit is targeting the correct branch (Camel 4 uses the `main` branch) # Tracking - [x] If this is a large change, bug fix, or code improvement, I checked there is a [JIRA issue](https://issues.apache.org/jira/browse/CAMEL) filed for the change (usually before you start working on it): [CAMEL-26967](https://issues.apache.org/jira/browse/CAMEL-26967). # Apache Camel coding standards and style - [x] I checked that each commit in the pull request has a meaningful subject line and body. - [ ] I have run `mvn clean install -DskipTests` locally from root folder and I have committed all auto-generated changes. (I built camel-kafka and the core modules with the formatter and import-sort plugins and ran the camel-kafka unit tests. I did not run the full root build.) # AI-assisted contributions - [x] If this PR includes AI-generated code, commits have proper co-authorship attribution (e.g., `Co-authored-by` trailers) and the PR description identifies the AI tool used. This PR was prepared with Claude Code (Claude Opus 5.5). The commit carries a `Co-Authored-By` trailer. _Claude Code on behalf of allthingssecurity_ 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
