adithyahm24 opened a new pull request, #18343: URL: https://github.com/apache/camel/pull/18343
# Description ## Cause - The batching logic in Camel-Kafka has a critical issue where records fetched in the current Kafka poll are skipped if the hasExpiredRecords method triggers processing due to the batchingIntervalMs condition. - These newly fetched records are not added to the exchangeList, and Kafka's polling behavior advances the offset, causing these records to be lost. - This results in data loss as unprocessed messages are neither processed nor retried in subsequent iterations. ## Fix ### Improved Timeout Handling: - Added logic to restart the timeoutWatch timer after processing expired batches. This ensures that the timeout is recalculated for subsequent records. ### Enhanced Record Addition Logic: - Introduced a clear separation between handling expired records and adding new records to the batch. This ensures that new records are always processed after handling any expiration. ### Batch Processing Adjustments: - Restarted the timeoutWatch timer after processing a batch when the maximum poll records limit is reached. This ensures accurate timing for subsequent batches. # Target - [ ] I checked that the commit is targeting the correct branch (Camel 4 uses the `main` branch) # Tracking - [ ] If this is a large change, bug fix, or code improvement, I checked there is a [JIRA issue](https://issues.apache.org/jira/browse/CAMEL-22150) filed for the change (usually before you start working on it). <!-- # *Note*: trivial changes like, typos, minor documentation fixes and other small items do not require a JIRA issue. In this case your pull request should address just this issue, without pulling in other changes. --> # Apache Camel coding standards and style - [ ] I checked that each commit in the pull request has a meaningful subject line and body. <!-- If you're unsure, you can format the pull request title like `[CAMEL-22150] Fixes bug in camel-file component`, where you replace `CAMEL-XXX` with the appropriate JIRA issue. --> - [ ] I have run `mvn clean install -DskipTests` locally from root folder and I have committed all auto-generated changes. <!-- You can run the aforementioned command in your module so that the build auto-formats your code. This will also be verified as part of the checks and your PR may be rejected if if there are uncommited changes after running `mvn clean install -DskipTests`. You can learn more about the contribution guidelines at https://github.com/apache/camel/blob/main/CONTRIBUTING.md --> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@camel.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
